The combination of dependency structures and part-of-speech (POS) tagging supplies us with the following information:
1. The linguistic category to which the word belongs
2. The words around it which are related to it
3. The type of that relation

An Idea
Merge many dependency structures into one big graph. The merge should be made by merging identical vertices. For example: the node ‘world’ across all the dependency graphs will be merged into a single vertex. This will create a pretty big graph with complex connection. In this graph we’ll focus on the nouns and extract them to be entities. Through looking at the other vertices that they are connected to (both other nouns and other parts of speech nodes), we can learn information regarding the entity that the noun represents. Since dependency the edges are labeled as well, we can even say something about the nature of these connections. It might be better to define a threshold of edges that will allow us to deduce a link. This way we can overcome accidental usage of a word, rare meanings that it has and simple errors.

For example, one thing we can do with this graph is to look only at the nouns and search for cycles in the graph. Such cycles can provide information about some kind of close relationship between the nouns in that cycle. Maybe it’s even possible to try to assess the nature of the relationship by looking at the path between two nouns.

Using this method we can build a huge database of entities with interesting applicative possibilities. For example: search engines can use such a database to improve search and the presentation of the search results for the user. A bit more far fetched possibility, but much more interesting, is the ability to actually learn something about the nature of languages through this. Furthermore, if we examine the structure we have as a graph, we can combine graphs from two language by connecting the entity nodes and use the web we receive.

Most information-rich websites, such as IMDB and AllMusic, display data in a highly structured template. This highly structured system can be enriched using meta-tagging of fields and relations between them. Similar to what you have in relational databases, but much richer and semantically oriented. If such meta-data is shared with search engines it can be used to create databases of entities, their attributes and relations between them. For example, if AllMusic will provide a link between an artist, which will be tagged as a ‘human’ to his ‘records’ using the connection ‘created’, it will be possible for search engines to read this data and use it to display results better or even harvest the web with partial semantical awareness.

In fact, I am quite sure search engines are already doing so to a certain extent. However, I am not familiar with a open and defined scheme that can describe this meta data and allow easy sharing of it. It has to be a flexible scheme that will allow to define various relationship (verbs).

The obvious usage for this method is in the field of information extraction. However, it can also prove useful in machine translation. Such mapped web sites will be much easier to translate. It can also help us to build a database of entities for research purposes. Something that can prove to be very useful in knowledge representation research. Therefore, for research, harvesting a few large sites and building entity databases from them, can prove very helpful and is quite feasible.

One of the interesting conclusion such an introspective action can lead us to is that the probabilistic direction, although not the only mechanism, is probably a step in the right direction. The way humans make mistakes at times, their fast deduction process, its result sometimes being altered after another second of thought and their tactics in handling ambiguity. All of these and many more linguistic behaviors point to a probabilistic approach in humans. However, as it seems (especially by the second-thought more accurate process), this approach is not the only mechanism and might be employed in an iterative fashion (answer quick and then try with slower a process to validate the answer).

One project which is really interesting in this case is the CHILDES corpus developed at CMU. Children don’t develop their language model by reading the WSJ10 corpus. Instead, they start with a very simple input and slowly develop their ability to handle more complex inputs. Since we know language acquisition in children to work quite well, we might be able to deduce from it a good start point for machine learning. If it’s simpler for children to acquire language in a gradual manner, it might also be the right way for computers.

The point is not to solve the human questions, but to use the information and logic revealed by a shallow examination. Later on, when there will be more information available, there will be more to copy.

The first principle is frequency. Basically it means that when faced with ambiguous input the subject should decide on which meaning to choose based on the past frequency of the various options. The option with the highest frequency will win. To optimize this principle one should pick the range of time properly. Meaning, the amount of time into the past that is used when calculating past frequencies of meanings. I believe this parameter, as used by the human mind, to be dynamic and at times results in a segment of time that is not even continuous. For example, one might understand a lecture better remembering what was said in the previous lecture two weeks ago and not what was said in the time between the lectures.

In one of my other posts here I talked about the idea of representing language through objects. Such objects, like linguistic items, will have to be linked and these links will have to reflect the ambiguity of the input. This gives us another option to over ambiguity - take the hottest links first. Meaning that one should chose the most recently used (MRU) path when coming to take links between objects. This method reflects the importance of the current situation on interpreting the input. It’s this second principle which I believe is actually of higher importance in daily life.

Our way of thinking is motivated by objects. It’s this understanding that lead to the creation of object oriented programming in the early 1960s. Therefore, it’s somewhat natural to deduce that the underlying data structures used by the brain for language processing might also consist of object oriented representation of our world. This, of course, ignores the more immediate problem of parsing and decoding. However, I still believe this to be theoretically interesting. I wish to suggest a very rough draft for such a system.

Let’s have for every type of object in the world a ‘class’, which is essentially the data type of this object. This is very similar to Aristotelian ideas (universal forms). This will allow the mind to hold data types with attributes and methods for each object in the universe. In a way, it’s a high level object oriented programming language. Each class will have components, which are basically members of other classes. For verbs we will create VerbTypes, which will be methods. Such methods will require parameters like the predicate and various optional arguments. For each verb we will have a verb type and each class will implement all of them. Such an implementation in a class will basically show what happens when you commit this verb on an object of this class. We will also have AdjTypes for each type of adjective (color, taste, smell and etc.). Again, each class will have attributes for each AdjType and these can be initialized or not. Also, a class will extend higher level classes.

To summarize, a class will look something like this:
Class Apple extends Fruit {
// Methods of type VerbType:
eat();
throw();
burn();
…

// Attributes of type AdjType:
color = …
taste = …
smell = …
…

// All sort of components:
Seeds
Peel
…
}

When creating a real world situation, the subject creates an inner representation of the outside situation. Also, it’s possible, once the grammatical structure has been chosen, to populate a sentence with objects through the relations created by the methods and their arguments. Thus we can construct sentences describing the happenings inside the virtual representation of the reality inside our minds.

This is, of course, only a rough sketch and it only comes to outline an attempt to deal with the problem of inner representation of language and real life situations. Naturally, it is far from complete and neglects many critical problems.

It’s not the whole language that is being hard-wired, as the big mass of learning (the vocabulary) is being made gradually over the years until the child reaches adulthood. However, it seems that the connections that form during this period enable better language acquisition and facilitate better learning of a certain language.

This system has two further effects:
1. It will make it relatively harder to acquire new language further on, especially when these differ greatly from the subject’s native language or even contradict it.
2. It will probably make it relatively easy to acquire languages based on similar founding principles. For example, it seems that German speakers have a relatively easy time learning English and French acquire Spanish with relative ease.

The idea that different languages bring to different wiring casts doubt upon the idea of universal grammar as advocated by Chomsky. Of course, it doesn’t mean that such a universal underlying structure doesn’t exist. It just means that if it does exist, it’s at very low level and the brain is actually wired according to a higher level logic. It also hints towards a gradual learning of a language, something many statistical models, which confront the learning machine with tons of data, somehow seem to miss.

It’s hard to say at what level the hardwired logic is. It’s definitely not the vocabulary of the language, probably not even its most basic parts. However, it’s something language specific. Maybe it’s the division into part of speech and the relation between these, or certain very basic grammatical structures and building blocks.

A few further note supporting this thesis:
In the few recorded cases of children who were isolated from society and weren’t given the opportunity to acquire a language, “post-liberation” language acquisition was never complete and their abilities didn’t reach these of normal children.

This approach to sifting out erroneous data is also being employed by humans when learning language. However, humans assist other factors in assessing the correctness of the input. Since a person has quite a bit of knowledge about his sources, he can judge their credibility and take it into consideration while processing the data originating from these sources. Our perception of people as credible or not is not binary (but fuzzy) and constantly develops. A quick introspection will reveal the complexity of the credibility system that we maintain and how it dynamically contributes to our linguistic learning over time.

Therefore, when coming to build a language acquisition model, it seems we should give the issue of source credibility its well deserved attention. Language acquisition in humans is linked to the sources of the data and to our ever-developing attitude towards them. Also, humans tend to acquire their first linguistic knowledge in a credibility pure environment (all sources are 100% credible), so it might be a good start for computer language acquisition as well.

Just a few points regarding this issue to highlight its complexity:
1. Not all sources are identical and a source’s properties dynamically change over time.
2. A certain input usually carries more weight when it comes from many sources. It seems that credibility is in some way accumulative over sources and time.
3. In some cases, even when the data is cast aside as false due to credibility problem, it still doesn’t disappear and might be reinstated and admitted into our linguistic system if and when the source’s credibility will update. However, usually this will happen only if the falseness of the data is ambiguous and the change in credibility happens shortly after the input is gained.