More flexible voting quorums Thread poster: Roland Nienerza
|
|---|
In the still open 9th Contest it appears again that the present scheme of the same fixed quorum of votes being required for all language pairings does not square with the actual resonance of the events in certain language combinations.
The organizers on the other hand do not hesitate to eliminate submissions in the qualifying stage if these submissions received consistent low ratings - even from a rather few voters. I was treated to that for one of my last entries on ratings of jus... See more In the still open 9th Contest it appears again that the present scheme of the same fixed quorum of votes being required for all language pairings does not square with the actual resonance of the events in certain language combinations.
The organizers on the other hand do not hesitate to eliminate submissions in the qualifying stage if these submissions received consistent low ratings - even from a rather few voters. I was treated to that for one of my last entries on ratings of just three voters.
In the present Contest there are still about 40% of the submissions unsettled after almost two months of final voting. In these almost two months there is consistently an outstanding number of 6 to 8 more votes for several combinations.
Why is it so difficult for the organizers to decide such cases on the tendency of the votes received - just as it is done on tendency of ratings in the qualification round? ▲ Collapse
| | | | | Language pairs are precisely being evaluated on a case-by-case basis | Dec 3, 2009 |
Hello Roland,
Roland Nienerza wrote:
The organizers on the other hand do not hesitate to eliminate submissions in the qualifying stage if these submissions received consistent low ratings - even from a rather few voters.
Please note that this is not so. During the qualification phase, entries are removed from contention when it becomes clear that they have no reasonable chance of winning. Such assessment is made based on the average rating received from peers and after a reasonable number of votes (just as in final voting rounds).
Roland Nienerza wrote:
Why is it so difficult for the organizers to decide such cases on the tendency of the votes received - just as it is done on tendency of ratings in the qualification round?
This is exactly what is being done now, Roland. As explained in previous posts, each language pair is evaluated separately, regardless of the tendency in other pairs. The purpose of this is to be sure that the winner entry in each pair has been consistently voted and absolutely deserves to win.
Hope this explains.
Regards,
Lucia
| | | | Laurent KRAULAND (X) 
France
Local time: 05:09
French to German
+ ...
Lucia Leszinsky wrote: Hello Roland, Roland Nienerza wrote:
The organizers on the other hand do not hesitate to eliminate submissions in the qualifying stage if these submissions received consistent low ratings - even from a rather few voters. Please note that this is not so. During the qualification phase, entries are removed from contention when it becomes clear that they have no reasonable chance of winning. Such assessment is made based on the average rating received from peers and after a reasonable number of votes (just as in final voting rounds).
Hello Lucia and Roland,
from my point of view, and speaking of my pairs, there is a problem about accepting and eliminating submissions (not that I did apply in any of these pairs). As an example, obvious machine translation entries were allowed... this in turn makes me wonder which criteria are required to be entering the contest.
| | | | | Any tool can be used for the translation of entries | Dec 3, 2009 |
Hello Laurent,
Please note that as contest FAQs state, there are no restrictions on tools used for the translation of contest entries.
Regards,
Lucia
| | |
|
|
|
| Why those MT entries are annoying - let's call a spade a spade | Dec 3, 2009 |
Lucia,
The issue here is not that some contestants used some tools to help them produce a translation. What we saw here in this round is that someone opened a profile at ProZ, registered in a tons of language pairs (112 pairs if I counted correctly) and submitted texts for the contest that were a direct output of Google translate. Yes, direct output, meaning the source text was simply fed into it and the raw output was submitted as a contest entry as is.
You should know exactl... See more Lucia,
The issue here is not that some contestants used some tools to help them produce a translation. What we saw here in this round is that someone opened a profile at ProZ, registered in a tons of language pairs (112 pairs if I counted correctly) and submitted texts for the contest that were a direct output of Google translate. Yes, direct output, meaning the source text was simply fed into it and the raw output was submitted as a contest entry as is.
You should know exactly which profile I am talking about, the profile owner did not hide its identity for the contest. Even if he/she did, as staff, you would still able to see it.
What really surprising is that this person purchased full membership for this prank!
Yes, I consider this a prank, and since there seems to be no other site activity from that profile, I would say this prank was its sole purpose.
As the contest organizer, you could probably see how many submissions came from this profile, and I am pretty sure you would come to the same conclusion.
One of these garbage texts made it into the final round in at least one pair (German-Hungarian) where there were only 3 submissions (and therefore the qualification round was skipped).
If you allow these sort of things happening, I think you are contributing to the factors that make the contests less professional.
Katalin ▲ Collapse
| | | | | a bit of numerics - | Dec 3, 2009 |
Roland Nienerza wrote:
The organizers on the other hand do not hesitate to eliminate submissions in the qualifying stage if these submissions received consistent low ratings - even from a rather few voters.
Lucia Leszinsky wrote:
Please note that this is not so. During the qualification phase, entries are removed from contention when it becomes clear that they have no reasonable chance of winning. Such assessment is made based on the average rating received from peers and after a reasonable number of votes (just as in final voting rounds).
I had understood rather lately, only in this present contest in fact, the difference in the voting during qualification, actually called "rating", and the final voting - being that in second case there is an accumulation of whole numbers so that even a low vote of just 1 point can make a difference, while in the first case, the qualification round the decision goes on the tendency of the average.
This difference of procedure is more than problematic, to put it mildly.
I give you an example.
Let us suppose that entry A gets in the qualification in each category four ratings, namely 4, 2, 1, 1 while entry B gets just one rating, a 3. This means that entry A has an average of 2 from 4 raters in either category and could be eliminated, while entry B has an average of 3 from 1 rater in either category and goes through to the final round.
But if both would have gone to the final on the basis of their absolute number of points - i.e. 16 for entry A and and 6 for entry B - and supposing that a similar voting pattern would appear in the final round, entry A with 8 points would have won single-handedly over entry B with 3 points. But with the current - more than problematic, to put it mildly - system, entry A will just not have reached the final round and B - in this simplified example - will get the prize.
The result of this is rather deplorable - and gives room for a lot of undesirable "tactics". - While in the final round a vote is vote, even if only for 1 point, a piece that collects - said with tongue in cheek - 120 times 1 point an amassing a nice total number of 120, will be as dead as the dodo and not even see the final round from far, although, with the current way of voting there, it might have won.
And here is a practical example of this present Contest, of a piece, I do not say from whom, eliminated in the qualification, I do not say of which pairing .with this result.
Rating, quality of writing: 2.33
Rating, accuracy of translation: 3.00
Number of raters: 3
Lucia Leszinsky wrote:
Such assessment is made based on the average rating received from peers and after a reasonable number of votes (just as in final voting rounds).
Do you, Lucia, call 3 a reasonable number of votes?
Roland Nienerza wrote:
Why is it so difficult for the organizers to decide such cases on the tendency of the votes received - just as it is done on tendency of ratings in the qualification round?
Lucia Leszinsky wrote:
This is exactly what is being done now, Roland. As explained in previous posts, each language pair is evaluated separately, regardless of the tendency in other pairs. The purpose of this is to be sure that the winner entry in each pair has been consistently voted and absolutely deserves to win.
Hope this explains.
Lucia
In does indeed, Lucia, at least to a certain point - and with the considerable reserves formulated above with regard to the, well, incoherences of the qualification round rating.
But why does this take almost two months?
[Edited at 2009-12-03 21:36 GMT]
| | | | | calling garbage a spade - | Dec 3, 2009 |
Katalin Horvath McClure wrote:
Lucia,
The issue here is not that some contestants used some tools to help them produce a translation. What we saw here in this round is that someone opened a profile at ProZ, registered in a tons of language pairs (112 pairs if I counted correctly) and submitted texts for the contest that were a direct output of Google translate. Yes, direct output, meaning the source text was simply fed into it and the raw output was submitted as a contest entry as is.
The somewhat remarkable thing about this is indeed the number. I would not even have guessed that Google has that many combinations in store. - But I would say that 112 is still only a fraction of the present number of contest submissions, and with the rate of one per pairing rather be of little importance.
Katalin Horvath McClure wrote:
One of these garbage texts made it into the final round in at least one pair (German-Hungarian) where there were only 3 submissions (and therefore the qualification round was skipped).
If you allow these sort of things happening, I think you are contributing to the factors that make the contests less professional.
Katalin
The topic has been discussed extensively in your thread, Katalyn.
It had been said by several posters that at least as of now little more is to be got from MT as what you call quite aptly garbage.
So, where is really the"annoying" quality about this? - Such submissions cannot really be voted into first place right now. - If they could - well, then we translators would definitely be in a fix.
I must say that I did not quite understand what you had called the "triangular" procedure, with a person procuring an MT from a source language unknown to him, but trimming that MT then into a submission for a target language he knows - maybe even his native language. - But this would actually be what Lucia has just mentioned, the use of a translation tool, as it can occur in the practice of the industry, and that should be permitted.
It had been proposed in your thread for this topic that a voting checkbox called "MT - yes / no" might be provided in future contests - and, if a piece gets a certain, maybe consistent number of yes-votes, it might be considered to exclude it.
[Edited at 2009-12-03 17:49 GMT]
| | | | | Roland, please do not drag the other thread in here | Dec 4, 2009 |
I appreciate that you remember the other thread. However, let's not drag that topic here, as that was different.
So, where is really the"annoying" quality about this? - Such submissions cannot really be voted into first place right now.
There are many reasons why I think it is annoying.
One of them is that it changes the situation in languages that have very few submissions. For instance, as the system works now, in a pair that has 2 legit submissions and one of these garbage entries, the qualification phase is skipped, the pair goes into final voting immediately. In reality, there are only 2 submissions competing. One of them will be the winner, no matter what. How does it compare to another language where there are also only 2 legit submissions, but no third garbage? That pair would probably get an extension and either get one ore more legit submissions, or closed without voting (nobody wins).
The only difference between the two pairs is whether the prankster targeted the pair, or not, however the outcome is different for the legit submissions.
Back to your posting regarding numerics:
I tried to follow the logic you described there - and I am not sure you understand how the voting goes in the final round.
As far as I remember, when you vote in the final round, first you have to mark one entry, that you think is the best. That will get 4 points. After you have done that, then you can mark the second best entry, and that gets 2 points. Finally, when you have marked the 1st and 2nd, you can mark the 3rd best, and that gets 1 point. You can stop after marking the best, and not mark anything for 2nd and 3rd, or you can stop after the 2nd, without marking the 3rd place. But you cannot mark a 3rd place without marking the 1st and 2nd place, the system simply does not let you do that. So, it is not possible for an entry to get 120 instances of 1 points without some other entries getting at least 120 instances of 4 points and 120 instances of 2 points.
I do agree with this:
But why does this take almost two months?
Katalin (with an i)
[Módosítva: 2009-12-04 03:04 GMT]
| | |
|
|
|
| Well, that makes indeed a difference - | Dec 4, 2009 |
Katalin Horvath McClure wrote:
As far as I remember, when you vote in the final round, first you have to mark one entry, that you think is the best. That will get 4 points. After you have done that, then you can mark the second best entry, and that gets 2 points. Finally, when you have marked the 1st and 2nd, you can mark the 3rd best, and that gets 1 point. You can stop after marking the best, and not mark anything for 2nd and 3rd, or you can stop after the 2nd, without marking the 3rd place. But you cannot mark a 3rd place without marking the 1st and 2nd place, the system simply does not let you do that. So, it is not possible for an entry to get 120 instances of 1 points without some other entries getting at least 120 instances of 4 points and 120 instances of 2 points.
Thank you for imparting me this instruction.
I have to admit that I only once attributed one vote for first place to another contribution. Because I normally check only pairings that I had submitted to myself and I got to say that in practically all cases I considered my own versions as best.
You are right that with the system as you describe it - and also to my recollection that seems indeed to be the way it works - an accumulation of votes for lower ranks could not happen without a multiple of that number for higher ranks.
And on that basis the elimination in the qualifying round of entries that get persistently low ratings would also make sense. With this explanation you have definitely ingratiated yourself not only with me - but also with Lucia, in sparing her the trouble to clarify this herself.
I hope that Lucia, who most probably will read this, may still think a bit about the example I quoted, with the elimination of a piece on the opinion exactly of 3 voters.
I appreciate that you remember the other thread. However, let's not drag that topic here, as that was different. So, where is really the"annoying" quality about this? - Such submissions cannot really be voted into first place right now. There are many reasons why I think it is annoying. One of them is that it changes the situation in languages that have very few submissions. For instance, as the system works now, in a pair that has 2 legit submissions and one of these garbage entries, the qualification phase is skipped, the pair goes into final voting immediately. In reality, there are only 2 submissions competing. One of them will be the winner, no matter what. How does it compare to another language where there are also only 2 legit submissions, but no third garbage? That pair would probably get an extension and either get one ore more legit submissions, or closed without voting (nobody wins). The only difference between the two pairs is whether the prankster targeted the pair, or not, however the outcome is different for the legit submissions.
With this we are back to contest quorums - though not so much with regard to voting as with regard to submission.
Because the prank you describe will make exactly and only a difference for pairings with just two serious - or legitimate, as you call them - entries. There, it is true, one could get a winner out of two contestants while without the mock submission as kicker there would not have been a vote at all. And - if this is what you call "annoying" - the weight of such a win does not compare with a win out of a much higher number of submissions.
Although you are focusing only on MT insertions. - I wonder whether you have not seen what I saw in combinations I entered, namely that occasionally there are submissions that are mind-boggling nonsense, although clearly not produced by or worked out on MT stuff, but concocted by real people, who are not working into their native tongue and assume skills they simply do not have. - I have even seen, and could name, a piece of that kind winning in a contest.
The problem of mock entry triggered votings are then rather the competition rules as they stand - allowing pairings with as little as 3 entries to get to a voting at all. As pranksters of the kind you mention are luckily rather rare - in fact you seem to target exactly one, albeit a very prolific one - they would hardly make a difference if the minimum quorum for submissions would be raised from 3 to whatsoever.
Not only your issue, Katalin, with what you called "immoral" and I call just "foolish or nonsensical" submissions, but also the issue of the strangely dragging out voting procedure would be solved in raising the "buy-in threshold quorum" for a pairings But as it stands, raising these quorums, let's say to ten, would shrink the number of pairings to 50% or less of what it presently is.
Katalin Horvath McClure wrote:
I do agree with this:
Roland Nienerza wrote:
But why does this take almost two months?
| | | | | What is still more astonishing.... | Dec 4, 2009 |
Laurent KRAULAND wrote:
As an example, obvious machine translation entries were allowed... this in turn makes me wonder which criteria are required to be entering the contest.
One of those (Italian-German) has got even 8 points! .
So I wonder: who on earth could give these votes? Certainly not German natives! It became obvious after the first sentence that this could not be a human translator's work. Facts like that doubtlessly don't second the seriousness of the whole thing. ProZ ist supposed to be a site for professional translators, isn't it?
In my opinion, the whole mechanism of the contest should be thought over: it is unbearable that after two months still the contest isn't closed and that texts like this one (and others) are admitted and even pass the finals round.
Furthermore, and apart from that, wouldn't it be possible to reintroduce the feature that showed, after the contest was closed, which text we voted for? After more than a month I think nobody is able to remember this. Why did you remove this feature that was very useful?
I for one have decided that for the time being and until things don't change I will never again participate in a contest (I did it once, in the beginning).
| | | | | One should not expect that organizers decide on admission. | Dec 4, 2009 |
Hello Christel,
we have to take it that this is a loosely organised contest to be decided by peer votes. It is not - simply cannot be - an event with submission to a jury of whatever composition, that first decides on acceptability of an entry, and then possibly also decides on the winners. It may be desirable to have such a jury, or juries for the different pairings. But they simply do not exist. And don't let us forget. Judging on translations involves so many little features tha... See more Hello Christel,
we have to take it that this is a loosely organised contest to be decided by peer votes. It is not - simply cannot be - an event with submission to a jury of whatever composition, that first decides on acceptability of an entry, and then possibly also decides on the winners. It may be desirable to have such a jury, or juries for the different pairings. But they simply do not exist. And don't let us forget. Judging on translations involves so many little features that one would have to expect a lot of discontent and wrangling about the decisions of such a jury too.
This answers largely the following questions -
Laurent KRAULAND wrote:
As an example, obvious machine translation entries were allowed... this in turn makes me wonder which criteria are required to be entering the contest.
Christel Zipfel wrote: One of those (Italian-German) has got even 8 points! . So I wonder: who on earth could give these votes? Certainly not German natives! It became obvious after the first sentence that this could not be a human translator's work. Facts like that doubtlessly don't second the seriousness of the whole thing. ProZ ist supposed to be a site for professional translators, isn't it?
But these questions raise a lot of disputable points in addition. ProZ.com is a platform allowing on the one hand many precious exchanges of people working in the industry and on the other hand it is essentially, at least as I understand it, a marketplace, where linguists offer service and outsourcers or other clients buy it.
The situation is similar, though of quite another dimension, to what I said above about contest admission. There simply is no jury that decides on admission to the platform - and I think that is alright. How come? -
Well, because the whole concept of a "professional" translator, though much used here and elsewhere in the industry, is not clearly defined. - One way to define it could be to reserve the term "professional" to graduates of dedicated translation training venues. - But what about a freshly graduated trainee of such an institution that did not know - or had forgotten - the declension of a core word like "heart" in his/her - I do not wish to be specific - mother tongue? - And in addition, such courses are far from being available for all language pairings, and sometimes are not affordable for some even if they exist.
Here comes the truth. - If there was a clear-cut definition for a "professional translator" one should expect, that two "professional translators" would translate the same source text, particularly of a limited size like in these contests, not only pretty much in the same manner, but above all in spotless, impeccable target language. - And then look into the contests. - Rather all over the board you will not only rarely find similarly construed solutions, you will rarely find, even among winning pieces!, linguistically flawless solutions. That much about "professional translators". -
In this sense, and I am glad to have seen this statement quite several times in other comments, the contest, for all its fun and entertainment, is quite a valuable kind of litmus test for competence of approach and precision of execution, two main criteria of professionalism.
The entry that you mentioned - in the pairing Italian-German - had been earmarked in the tagging several times as unacceptable, and pinpointed as most probably being Machine Translation, with at best little editing. And you rightly wonder how this could get 8 points - even 6 more than a clearly better "human" piece.
I too had wondered with regard to the winner of another combination in this contest - as about the winner in one of the earlier contest - who on earth could be the people who have voted such pieces into such positions.
I definitely would like to ask whether the original stance to keep the voting anonymous is really to the purpose. - After all, names of raters are now displayed too, after the final voting. And even in KudoZ, the agrees and disagrees are given with names. - ▲ Collapse
| | | | To report site rules violations or get help, contact a site moderator: You can also contact site staff by submitting a support request » More flexible voting quorums | Anycount & Translation Office 3000 | Translation Office 3000
Translation Office 3000 is an advanced accounting tool for freelance translators and small agencies. TO3000 easily and seamlessly integrates with the business life of professional freelance translators.
More info » |
| | CafeTran Espresso | You've never met a CAT tool this clever!
Translate faster & easier, using a sophisticated CAT tool built by a translator / developer.
Accept jobs from clients who use Trados, MemoQ, Wordfast & major CAT tools.
Download and start using CafeTran Espresso -- for free
Buy now! » |
|
| | | | X Sign in to your ProZ.com account... | | | | | |
Login to reply/comment 
