投稿
APP
微信扫一扫获取更多

疾速进入自然言语处理NLP

打了个飞机

2018-12-16 11:40:47

图灵汇官网

自然语言处理简介

计算机擅长处理结构化数据，比如数据库表格和财务记录，这使得它们在处理这类数据时比人类快得多。然而，人类交流并不依赖于结构化数据，而是使用文字，这是一种典型的非结构化数据。

不幸的是，计算机处理非结构化数据的能力有限，因为目前没有标准化的方法来处理这种数据。当我们用编程语言（如C、Java或Python）对计算机编程时，实际上是给出了一系列规则，但对于非结构化数据，这些规则往往过于抽象且难以具体定义。

人类与计算机对语言的理解

人类写作已经有几千年的历史，这期间我们的大脑积累了大量理解自然语言的经验。当我们阅读书本或网上的博客时，能理解其在现实世界中的实际含义，甚至能够感受到文字背后的情感，并在脑海中构想出相应的场景。

自然语言处理（NLP）是人工智能的一个分支，致力于让计算机能够理解和处理人类的语言，使计算机更接近人类对语言的理解。尽管如此，计算机并不能像人类一样理解语言的深层含义。简而言之，计算机无法“读懂”字里行间的意思。

然而，近年来机器学习的发展使得计算机可以通过自然语言完成许多有用的任务。深度学习技术使我们能够编写程序来进行诸如语言翻译、语义理解和文本摘要等任务，从而极大地提升了效率。

NLP面临的挑战

理解和解析语言的过程远比表面上看起来复杂得多。要真正理解一段文字在现实世界中的含义，需要考虑许多因素。例如，你认为下面这句话意味着什么？

"Steph Curry昨晚表现神勇，彻底击溃了对手。"

对人类来说，这句话的意思很清楚。我们知道Steph Curry是一位篮球运动员，他昨晚的表现非常出色，击败了对手。

计算机则倾向于逐字逐句地理解。从字面上看，“着火”和“击溃”可能被理解为字面意思，计算机可能会误解为Steph Curry真的着火了。但通过机器学习，我们可以利用一些简单的Python库来快速提取和理解信息。

使用Python处理NLP问题

为了更好地理解NLP的工作原理，我们将使用Wikipedia中的文本作为示例：

Amazon.com, Inc., doing business as Amazon, 是一家位于美国西雅图的电子商业和云计算公司，由Jeff Bezos于1994年7月5日创立。作为科技巨头，亚马逊是全球最大的互联网零售商，按收入和市值衡量。亚马逊网站最初是一家在线书店，后来扩展到销售视频下载/流媒体、MP3下载/流媒体、有声读物下载/流媒体、软件、视频游戏、电子产品、服装、家具、食品、玩具和珠宝。公司还生产消费电子产品，如Kindle电子阅读器、Fire平板电脑、Fire TV和Echo，并且是全球最大的云基础设施服务提供商（IaaS和PaaS）。亚马逊还销售一些低档产品，品牌为Amazon Basics。

需要的库

首先，我们需要安装一些有用的Python NLP库： bash pip3 install spacy python3 -m spacy download en_core_web_lg pip3 install textacy

实体分析

现在所有库都已安装完毕，我们可以对文本进行实体分析。实体分析会遍历文本并识别其中的重要词汇或“实体”。我们真正关心的是那些具有现实世界语义意义的单词。

下面是实体分析的代码： ```python import spacy

加载spaCy的英语NLP模型

nlp = spacy.load('encoreweb_lg')

我们要分析的文本

text = """ Amazon.com, Inc., doing business as Amazon, is an American electronic commerce and cloud computing company based in Seattle, Washington, that was founded by Jeff Bezos on July 5, 1994. The tech giant is the largest Internet retailer in the world as measured by revenue and market capitalization, and second largest after Alibaba Group in terms of total sales. The amazon.com website started as an online bookstore and later diversified to sell video downloads/streaming, MP3 downloads/streaming, audiobook downloads/streaming, software, video games, electronics, apparel, furniture, food, toys, and jewelry. The company also produces consumer electronics - Kindle e-readers, Fire tablets, Fire TV, and Echo - and is the world's largest provider of cloud infrastructure services (IaaS and PaaS). Amazon also sells certain low-end products under its in-house brand AmazonBasics. """

解析文本

document = nlp(text)

输出所有检测到的实体

for entity in document.ents: print(entity.text, entity.label_) 运行这段代码会输出一系列实体及其标签，例如： Amazon.com, Inc. ORG Amazon ORG American NORP Seattle GPE Washington GPE Jeff Bezos PERSON July 5, 1994 DATE second ORDINAL Alibaba Group ORG amazon.com ORG Fire TV ORG Echo ORG PaaS ORG Amazon ORG AmazonBasics ORG ```

从输出可以看出，我们的模型做得不错。Jeff Bezos被识别为一个人，日期是正确的，亚马逊被识别为一个组织，西雅图和华盛顿也被识别为地缘政治实体。不过，Fire TV和Echo被误认为是组织而非产品，但这是可以接受的误差范围。

总的来说，这个模型已经完成了我们所需要的任务。想象一下，如果我们有一个包含几百页文本的大文档，这个NLP模型可以帮助我们快速了解文档内容及文档中的关键实体。

对实体进行操作

假设我们有一些文本块，出于隐私考虑，希望自动删除所有个人和组织名称。spaCy库提供了一个非常有用的清除函数，我们可以用它来清除任何我们不想看到的实体类别： ```python import spacy

加载spaCy的英语NLP模型

nlp = spacy.load('encoreweb_lg')

要分析的文本

定义替换实体的函数

def replaceentitywithplaceholder(token): if token.entiob != 0 and (token.enttype == "PERSON" or token.enttype == "ORG"): return "[PRIVATE] " else: return token.string

清除文本中的实体

def scrub(text): doc = nlp(text) for ent in doc.ents: ent.merge() tokens = map(replaceentitywith_placeholder, doc) return "".join(tokens)

print(scrub(text)) ``` 这段代码会将文本中的个人和组织名称替换为“[PRIVATE]”，从而保护隐私。

从文本中提取信息

我们之前安装的textacy库在spaCy的基础上实现了多种NLP信息提取算法。它允许我们执行一些更高级的任务，比如半结构化语句提取。这个算法可以基于spaCy的NLP模型提取一些关于特定实体的更详细的信息。

下面是一个例子，我们将提取关于华盛顿特区的一些信息： ```python import spacy import textacy.extract

加载spaCy的英语NLP模型

nlp = spacy.load('encoreweb_lg')

要分析的文本

text = """ Washington, D.C., formally the District of Columbia and commonly referred to as Washington or D.C., is the capital of the United States of America. Founded after the American Revolution as the seat of government of the newly independent country, Washington was named after George Washington, first President of the United States and Founding Father. Washington is the principal city of the Washington metropolitan area, which has a population of 6,131,977. As the seat of the United States federal government and several international organizations, the city is an important world political capital. Washington is one of the most visited cities in the world, with more than 20 million annual tourists. The signing of the Residence Act on July 16, 1790, approved the creation of a capital district located along the Potomac River on the country's East Coast. The U.S. Constitution provided for a federal district under the exclusive jurisdiction of the Congress and the District is therefore not a part of any state. The states of Maryland and Virginia each donated land to form the federal district, which included the pre-existing settlements of Georgetown and Alexandria. Named in honor of President George Washington, the City of Washington was founded in 1791 to serve as the new national capital. In 1846, Congress returned the land originally ceded by Virginia; in 1871, it created a single municipal government for the remaining portion of the District. Washington had an estimated population of 693,972 as of July 2017, making it the 20th largest American city by population. Commuters from the surrounding Maryland and Virginia suburbs raise the city's daytime population to more than one million during the workweek. The Washington metropolitan area, of which the District is the principal city, has a population of over 6 million, the sixth-largest metropolitan statistical area in the country. All three branches of the U.S. federal government are centered in the District: U.S. Congress (legislative), President (executive), and the U.S. Supreme Court (judicial). Washington is home to many national monuments and museums, which are primarily situated on or around the National Mall. The city hosts 177 foreign embassies as well as the headquarters of many international organizations, trade unions, non-profit, lobbying groups, and professional associations, including the Organization of American States, AARP, the National Geographic Society, the Human Rights Campaign, the International Finance Corporation, and the American Red Cross. A locally elected mayor and a 13-member council have governed the District since 1973. However, Congress maintains supreme authority over the city and may overturn local laws. D.C. residents elect a non-voting, at-large congressional delegate to the House of Representatives, but the District has no representation in the Senate. The District receives three electoral votes in presidential elections as permitted by the Twenty-third Amendment to the United States Constitution, ratified in 1961. """

解析文本

document = nlp(text)

提取半结构化语句

statements = textacy.extract.semistructured_statements(document, "Washington")

输出提取的信息

print("* 从华盛顿特区的维基百科页面获取的信息 *") count = 1 for statement in statements: subject, verb, fact = statement print(f"{count} - 陈述: {statement}") print(f"{count} - 事实: {fact}") count += 1 ``` 通过这段代码，我们提取到了关于华盛顿特区的几个关键信息： 1. 华盛顿是美国的首都。 2. 华盛顿的人口及其在大都会区的地位。 3. 许多国家纪念碑和博物馆。

这些信息正是这段文本中最重要和最有价值的部分。