Following the assumption of hardware independence, the analysis of the running time of an algorithm depends only on the statements defined in the algorithm (i.e., the steps that the algorithms need to follow for solving a specific problem). Since the execution time of the algorithms’ statements is not the same for all of them (i.e., some steps need more time to be executed than others), it is necessary to be defined a set of rules.

Rule 1:

The time required to fetch an integer from memory is defined as t_fetch. The time needed to store an integer to the memory is defined as t_store. 

For example, the running time of the statement “x = 1” is t_fetch + t_store, because the integer is in the memory (typically any constant is stored in memory) and have to be fetched and also this integer has to be stored to the variable “x” which is in memory. The same running time has the statement “x = y” because we want to get the value of the variable “y” which is stored in memory.   

Rule 2:

The time needed for each basic operation to be executed is defined as t₊, t_–, t_x, t_<, t_/. So, the running time of the statement “y = x + 1” is 2t_fetch + t₊ + t_store. The 2 fetches are for “x” and “1”. The result is stored in variable “y” (t_store). The addition operation time is t₊.  

Rule 3:

The time needed to a function to be called is defined as t_call while the time required to a function to return a value is defined as t_return.

Rule 4:

The time needed to pass a value as parameter to a function is the same as the time needed to store a value in memory. For example, the running time of “y = f(x)” is 2t_store + t_fetch + t_call + T_f(x). Note here that the t_return is part of the function and not of the above statement. Furthermore, the T_f(x) is the total running time of the function “f”.   

Rule 5:

The time required for the address calculation implied by an array subscripting operation is defined as t_[] . For example, the running time of the statement “y = a[i]” is 3t_fetch + t_store +t_[]. The fetches are (1) the value of i, (2) the value of a and (3) the value of a[i].  

Using these rules, we can easily estimate the running time of the algorithms.

Examples

Here, we present some programs in C++ and we estimate their running time.

The running time of the function “max” depends on the lines 2,3 and 4. So, we have to estimate for each line (or step) the corresponding time and then to calculate the total time of the function.

Making our life better, we define the total time of max function when “a > b” as t₁ and the total time of max function when “a <= b” as t₂. We can say that T_max(a,b) = t₁  ή T_max(a,b) = t₂. However, as we can see  t₁ = t₂, so the T_max(a,b) = 3t_fetch + t_< + t_return. if t₁ <> t₂, then the total worst running time of the function would be that t with the worst time.

The function “findMaxNumber” gets as input an array A with n values and returns the maximum number of the array. We also use the previous function “max” to find the maximum number of two variables. The running time of the function depends on the lines 2 to 6.

As we can see in the table, we used the 5 rules defined above to find the running time of the algorithm (Exercise: Find the total running time of the algorithm using also the running time of the function “max”). There are some interesting parts in this source code:

• The line 3 is split into three parts. The initialization of the i variable is executed only one time while the statements 3b and 3c are executed at least n times. The 3b statement is executed n + 1 times because we also take into account when the condition is getting false for the first time.
• To find the running time of the 4th statement we needed all 5 rules. Firstly, we have to fetch the A[i] (as we explained above) and the maxNumber. These two parameters need 2t_store when they are passed to the function. The return value of the function is stored in the maxNumber so we need also a t_store.

The post Estimating Running Time of an Algorithm using Simple Rules. appeared first on Dimitris Dimitriadis.

• Q: a finite set of N states \(q_0, q_1,…,q_N\)
• Σ: a finite input alphabet of symbols
• \(q_0\): the start state
• F: the set of final states, \( F \subseteq Q \)
• δ(q,i): the transition function or transition matrix between states. Given \(q \in Q\) and an input symbol \( i \in Σ \), δ(q,i) returns a new state \(q’ \in Q\). δ is thus a relation from \( Q \times Σ \to Q \);

Using this machine, one can recognize a formal language. So, if we define a formal language with a regular expression, an automaton will be in a position to recognize it.

In this post, we will work with two formal languages to show the algorithms that can be used for recognizing the languages by finite state automaton. We will implement the algorithms in python. You can find more information about the use of deterministic finite state automata here. Furthermore, in this post one has to know what we mean “tape” and “transition matrix/table”. For regular expressions, please visit here.

The Sheep Talks (character-level tapes)

Our first example is a simple formal language for the sheep language. The sheep language is any string of the form:

• baa!
• baaa!
• baaaa!
• ….

The corresponding regular expression is ^baa+!$. So, we want a finite state automaton to recognize this regular expression.

So, following the parameters defined above, \(Σ = \lbrace a,b,! \rbrace \) while for the states and transitions we will use a transition matrix. We define these elements in python

The transition matrix contains the transition between two states given a letter. For example, given the letter ‘b’ in the \(q_0\) state, we can pass to the state \(q_1\). In the \(q_1\) the FSA accepts the letter ‘a’ and goes to the state \(q_2\) and so on. The None values representes the invalid characters for a given state. So, if in state \(q_0\) the letter ‘a’ will be given, then the given string (or tape) is invalid. Next we define a function that will be necessary later:

This function get as input the current symbol of a string (or tape) and the list of all available symbols. The aim of the function is to convert the letters to numeric values that corresponds to the collumns of the transition matrix. For example, if the given string is “baba!” the corresponding output will be “01012”. Next we define the algorithm that recognizes a given tape.

This function get as input the tape, the transition matrix the symbols and the accepted states (F set as defined above). The aim of the function is to pass through all letters and to decide if the given tape is valid. Particulalry, it uses the below conditions:

• if we are at the end of the tape, then if the state is final, then the given tape is accepted, otherwise is not.
• if we are not at the end of the tape, we check if the current letter in the current state can be considered as valid (i.e. not None). If the letter is not valid then the tape is rejected. Otherwise, we go to the next state based on the transition matrix and we update the index that points to the next letter in the tape.

To test our algorithm, we define some tapes and we call the above function

A more complex example – dollars and cents (word-level tapes)

Now, suppose that we want to recognize money with our FSA. For example

• twenty five dollars
• five dollars thirty cents
• five cents
• ……

We will work as previously but here the transition matrix is bigger due to the set of symbols.

Here the symbols are all the words that corresponds to the numeric values (we ignore cases with hundreads and thousands for simplicity). The transition matrix consists of 7 states, while the states that are full of Nones mean that they are final states.

Defining the bellow strings, we will see the expected results

Bibliography

• Jurafsky, D. (2000). Speech & language processing. Pearson Education India.

The post Deterministic Finite State Automota for Regular Expressions appeared first on Dimitris Dimitriadis.

1. Gmail: The email of the university members have the prefix “auth.gr”
2. Google Drive: Unlimited free space
3. Google Calendar
4. Google Docs, Sheets, Slides and Forms
5. Google Classroom
6. Google Sites
7. Google Hangouts
8. Blogger, Google Groups and Google+

To learn more about the services, visit the https://it.auth.gr/el/cloudServices/gapps. To enable the service, visit https://accounts.auth.gr/misc/googleApps.php.

To use the service, one has to wait 8 hours for the activation. Later, he has to be signed in to google services using the email username@gapps.auth.gr

For any questions, feel free to contact with me.

The post Google Apps for Education (Aristotle University of Thessaloniki) appeared first on Dimitris Dimitriadis.

As we can see, there are plenty of special cases in QA, thus, it would be beneficial to find the datasets that can help us to train efficient models on a particular QA task.

The aim of this article is to present these datasets giving a description for each of them and also links to download them and experimented with them.

Natural Questions

This dataset has been constructed in 2019 by the Google Research team for extractive QA. The dataset contains thousands of queries that have been made by users in the Google search engine. Particularly, the dataset is split into the train, dev, and test set, with approximately 310,000, 8000, and 8000 instances accordingly. An instance is a question (user query), a Wikipedia page from the top 5 search results, a long answer which has the length of a paragraph, and a short answer which is one or more entities or yes/no [1]. The Wikipedia page is in the HTML format while the dataset is in the JSON Lines text format. (In Python, one can use the jsonlines library to process the dataset). The dataset is available here.

Squad V1.0

This is another dataset for extractive QA constructed by Stanford Research Team. Crowdworkers posed questions on a set of Wikipedia articles, while the answer to the questions is a span of text included in a passage contained in the Wikipedia articles. The number of questions is 100k approximately [2]. The human performance on the dataset is 86.8%. The dataset is available here.

Squad V2.0

The previous version of the dataset contained easy questions. Thus the state-of-the-art models achieved better results even from humans. In detail, the main issue of the past state-of-the-art models was that they tried to guess the answer in the context of a passage, while the answer was not stated in this context. Stanford observed that and introduced the second version of the dataset [3]. The dataset contains the questions from version 1 plus the 50k unanswerable questions. Now, the models must answer the questions only if it is possible. Based on the leaderboard, the best accuracy is 89% overcoming human judgment. The dataset is available here.

BoolQ

This dataset contains yes/no questions. Particularly, the creators of the dataset gathered questions from other datasets such as Natural Questions or constructed triples of questions, passages, and answers using other sources to build a dataset only with yes/no questions. They claim that to answer the questions, one must infer the answer from the passages while reasoning is mandatory. They introduced six types of reasoning. Their best model was one that trained on Natural Language Inference dataset and fine-tuned on the boolq dataset [4]. The dataset is available here.

BioASQ

A biomedical domain QA dataset which was constructed by biomedical experts for the BioASQ Challenge dated back to 2013. The dataset contains questions of several types (yes/no, factoid, list, summary), while the systems should answer a question giving a short answer (exact answer) and a paragraph-like answer (ideal answer) [5]. The dataset is available here. Note that if one wants to use the dataset, he must also register in the platform.

Bibliography

[1] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., … & Toutanova, K. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, 453-466.

[2] Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

[3] Rajpurkar, P., Jia, R., & Liang, P. (2018). Know What You Don’t Know: Unanswerable Questions for SQuAD. arXiv preprint arXiv:1806.03822.

[4] Clark, C., Lee, K., Chang, M. W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019). BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv preprint arXiv:1905.10044.

[5] An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition: George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel Ngonga, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos and Georgios Paliouras, in BMC bioinformatics, 2015

The post Datasets In Question Answering appeared first on Dimitris Dimitriadis.