8 – NLP M1-L1 01 NLP Pipeline

Let’s look at a common NLP pipeline. It consists of three stages, text processing, feature extraction and modeling. Each stage transforms text in some way and produces a result that the next stage needs. For example, the goal of text processing is to take raw input text, clean it, normalize it, and convert it into a form that is suitable for feature extraction. Similarly, the next stage needs to extract and produce feature representations that are appropriate for that type of model you’re planning to use and the NLP task you’re trying to accomplish. When you’re building such a pipeline, your workflow may not be perfectly linear. Let’s say, you spend some time implementing text processing functions, then make some simple feature extractors, and then design a baseline statistical model. But then, maybe you are not happy with the results. So you go back and rethink what features you need, and that in turn, can make you change your processing routines. Keep in mind that this is a very simplified view of natural language processing. Your application may require additional steps.

%d 블로거가 이것을 좋아합니다: