Leaving the Frontiers
On 14 August 2014 I was invited to become an associate editor (AE) of the Open-Access Frontiers in Human-Computer Interaction journal (part of the 'Nature' publishing group)
I support Open Access, and while I was curious what this collaboration with Nature could do for it, I have become increasingly uneasy being associated with Frontiers. Nature itself documented how the journal is now added to Beal's List of questionable publishers. There were significant issues with the editorial independence of Frontier's medical section, written up well by Leonid Scheider. The same author also penned a piece describing the problems with interns at Frontiers.
But in particular, I feel the peer review process is toothless and that it is entirely setup to effectively act as a pay-to-publish model. That some of Frontier's sections also publish good papers is a matter of luck, in my opinion. I only supported the review process of a couple of papers, mostly because almost none of the papers submitted to this journal were close to my area of expertise. But in both cases I felt there was no way in which the reviewers' comments could ever lead to the paper not being accepted. In fact, the entire peer review process, including its web-interface, seems geared towards slowly but surely progressing a manuscript to publication. The official Associate Editor guidelines issued by Frontiers say as much: reviewers should not gauge a paper on the basis of 'novelty', but on 'correctness'. The idea is that posterity will signify a paper's significance, presumably measured by e.g. citation counts. This basically means that anything can get published, as long as you correct your errors and persevere. The insistence that reviewers respond to each new comment of authors turns it into a match of who gives up first. Persistent authors will always be able to publish. Provided they pay Frontiers, of course.
While there are many things wrong with the standard peer review process, like Democracy, I think it's the least bad system we have. Yes, it's based on a limited number of experts. And yes, the process is unduly lenghty, un-interactive and the comments and demands of reviewers (and AEs) can sometimes be downright petty. But, in the end, I believe that it provides readers iwth a solid form of trust that a paper is correct, and worthwhile to read. Good journals will be able to call upon better reviewers and AEs, and as such the whole process is more solid for the more elite journals as well, which is also good.
In this new era of populism and fake news stories circulating on social media, I believe going down the Frontiers route that basically says 'if it's mentioned/cited/liked by people, it must be good' is simply the wrong way to go. I believe in the value of expert reviews, and I believe in the power of a trust-based society. Therefore I am today standing down as an Associate Editor for Frontiers.
I support Open Access, and while I was curious what this collaboration with Nature could do for it, I have become increasingly uneasy being associated with Frontiers. Nature itself documented how the journal is now added to Beal's List of questionable publishers. There were significant issues with the editorial independence of Frontier's medical section, written up well by Leonid Scheider. The same author also penned a piece describing the problems with interns at Frontiers.
But in particular, I feel the peer review process is toothless and that it is entirely setup to effectively act as a pay-to-publish model. That some of Frontier's sections also publish good papers is a matter of luck, in my opinion. I only supported the review process of a couple of papers, mostly because almost none of the papers submitted to this journal were close to my area of expertise. But in both cases I felt there was no way in which the reviewers' comments could ever lead to the paper not being accepted. In fact, the entire peer review process, including its web-interface, seems geared towards slowly but surely progressing a manuscript to publication. The official Associate Editor guidelines issued by Frontiers say as much: reviewers should not gauge a paper on the basis of 'novelty', but on 'correctness'. The idea is that posterity will signify a paper's significance, presumably measured by e.g. citation counts. This basically means that anything can get published, as long as you correct your errors and persevere. The insistence that reviewers respond to each new comment of authors turns it into a match of who gives up first. Persistent authors will always be able to publish. Provided they pay Frontiers, of course.
While there are many things wrong with the standard peer review process, like Democracy, I think it's the least bad system we have. Yes, it's based on a limited number of experts. And yes, the process is unduly lenghty, un-interactive and the comments and demands of reviewers (and AEs) can sometimes be downright petty. But, in the end, I believe that it provides readers iwth a solid form of trust that a paper is correct, and worthwhile to read. Good journals will be able to call upon better reviewers and AEs, and as such the whole process is more solid for the more elite journals as well, which is also good.
In this new era of populism and fake news stories circulating on social media, I believe going down the Frontiers route that basically says 'if it's mentioned/cited/liked by people, it must be good' is simply the wrong way to go. I believe in the value of expert reviews, and I believe in the power of a trust-based society. Therefore I am today standing down as an Associate Editor for Frontiers.
The problem with monetarising academic science
26/05/11 10:39 Filed in: Research
On Tuesday the 24th of May, I attended the Royal Institution’s event titled ’Science careers: has the science establishment let down young researchers?’. Dr Evan Harris chaired a panel discussion on careers in science at the Royal Institution. On the panel were Dr Jenny Rohn and Professor Dame Athene Donald, FRS, and they were joined by Rt Hon David Willetts MP, Minister of State for Universities and Science. The event was masterfully led by Dr Evan Harris, who made use of the late running of the minister to make a survey of what topics the audience wanted to discuss, and to make a selection of what questions and opinions should definitely be put before Willetts.
Many issues were raised, ranging from the two body problem induced by the pressure on postdocs to gather experience at different universities, to the leaky pipe syndrome experienced by female scientists for whom the lack of job security is a problem if they are planning for a family. Perhaps the biggest problem for young scientists to advance their career, i.e. to become a PI, is the current structure of the scientific community. The ratio of PIs to postdocs and PhDs is sometimes described as a pyramid, but as Dr Jenny Rohn put it, it’s more like a big spike in the middle of a flat surface. That is a nice property for a signal in machine learning, but it’s not a desirable property for your career prospects. A recent study by the RA showed that only 4% of PhDs ended up in an academic position. The minister accepted that such problems existed, but made it clear that in the government’s opinion such problems should be addressed by the academic community, and not regulated by government.
While I agree with Willetts that it is the academic community that needs to change its ways, I do think there’s a great role for the government to play, albeit not necessarily by setting up new laws and regulations. But I will get back to that at the end of my discussion. In my opinion, the fundamental problem is with how academic research is monetarised. That is, how do we value academic research and how do we then proceed to distribute research funding. Funding requests are proposed by an individual or a small group of individuals. At the moment, the prevailing measure for research quality of these individuals is their publication record. Someone with a lot of publications and citations must be a good researcher, and is thus worthy of the funding requested.
This mentality, if not properly constrained, generates groups where a single PI hires as many PhDs and PostDocs as possible, who do the actual research described in the funding proposals, and publish papers on which the PI is invariably a co-author. This increases the research quality of the PI, resulting in more funding, resulting in more PhDs and PostDocs, resulting in more papers... you get the picture. There is no incentive in this mechanism not to have a big group of low-paid eager employees led by a single PI. It simply works.
So what’s wrong with this picture? A few things, I think. While this mechanism may function, it does not necessarily generate the best possible research for the money spend, and may thus not be in the best interest of the country and the government. Why not? First of all, both PhDs and PostDocs are supposed to be in training by the PI. PhDs more so than PostDocs, but even they are supposed to receive training as a PostDoc is supposed to be a preparatory stage before becoming a lecturer. It’s not hard to see that the more spiky the employment structure, i.e. the more PhDs/PostDocs a PI has, the lower the quality of this training, and thus the lower the quality of the resulting research. Spiky structures also mean low career prospects, and this has a huge demoralising effect on the very people expected to conduct the actual research. Yet there is no incentive in the current monetarising mechanism to limit the number of PhDs/Postdocs for each PI, in fact, the opposite is encouraged.
Secondly, it is reasonable to assume that the best researchers end up being PIs, eventually. Yet in the current system, all their time is consumed in writing grant proposals, and managing their research groups. This is a waste of their potential: they are supposed to be the best researchers available to the community yet they cannot perform any actual bench work! One argument often heard against this is that they at least still generate the ideas that form the basis for grant proposals. Yet it is a well known fact that proposals do not consist of entirely novel ideas. Because of the harsh reviews governments and research councils put in place these days, PIs promise to research things that have already been attained, or are at least within short reach. So many research proposals are not stellar new proposals for groundbreaking research, but bland continuations of existing research, designed to have easily evaluated performance measures to suit the bureaucratic reviews.
And that forms also the basis for the third issue of spiky research groups. While PostDocs are supposedly independent researchers, mentored by their professor, in reality they are performing the research set out by their PI and formulated in part by the grant proposals. This means that they are not fully able to pursue their own ideas, which can be truly paradigm shifting, and again a lot of potential research excellence is lost.
So, how can we go about changing this in a structured manner, that is, by changing the actual mechanism of monetarising academic research? I see two things that can be done by the scientific community. First of all, universities can set a limit to the number of PhDs/PostDocs that a PI can have, based on their obligation to properly supervise. This would ensure that PhDs and PostDocs are properly trained, and that would in turn improve their research output. Universities may be afraid of this because it would limit their star funding attractors, but this is not an objection if they would increase the ratio of PIs to PostDocs, which is the principal problem we are addressing anyway, a problem which I hope the universities acknowledge exists.
Secondly, universities and funding bodies should make it clear that it is ‘not done’ to be a co-author on a paper merely on the merit of having procured the funding for a subset of the authors. Some funding bodies in other countries (e.g. Germany) already make this explicit in their funding rules, and increasingly prominent journals and conferences ask the authors to declare that all authors have made a significant contribution to the research described in the paper (not the research funding). If these guidelines were upheld, they would force PIs to spend more time on actual research for fear of losing their measured research quality, which would mean they would have less time to manage huge groups and write loads of funding proposals, which in turns makes place for more PIs with better structured (i.e. less spiky) groups.
One obstacle remains: recently there has been a push by funding bodies to reduce the number of grant proposals coming in, as a cost cutting measure. The proposals I made above will both lead to an increase of proposals, at least there will be more proposals by more people. This is a good thing though, and the fallacy of reducing funding requests must be tackled. This is one thing where Willetts can act: provide a larger percentage of the budget to the research councils for dealing with grant proposals.
The other way in which Rt. Hon. Willetts can influence this process is by influencing universities and research councils that a de-spiking of the research community structure is a good thing, and that the ideas outlined above should be supported.
Many issues were raised, ranging from the two body problem induced by the pressure on postdocs to gather experience at different universities, to the leaky pipe syndrome experienced by female scientists for whom the lack of job security is a problem if they are planning for a family. Perhaps the biggest problem for young scientists to advance their career, i.e. to become a PI, is the current structure of the scientific community. The ratio of PIs to postdocs and PhDs is sometimes described as a pyramid, but as Dr Jenny Rohn put it, it’s more like a big spike in the middle of a flat surface. That is a nice property for a signal in machine learning, but it’s not a desirable property for your career prospects. A recent study by the RA showed that only 4% of PhDs ended up in an academic position. The minister accepted that such problems existed, but made it clear that in the government’s opinion such problems should be addressed by the academic community, and not regulated by government.
While I agree with Willetts that it is the academic community that needs to change its ways, I do think there’s a great role for the government to play, albeit not necessarily by setting up new laws and regulations. But I will get back to that at the end of my discussion. In my opinion, the fundamental problem is with how academic research is monetarised. That is, how do we value academic research and how do we then proceed to distribute research funding. Funding requests are proposed by an individual or a small group of individuals. At the moment, the prevailing measure for research quality of these individuals is their publication record. Someone with a lot of publications and citations must be a good researcher, and is thus worthy of the funding requested.
This mentality, if not properly constrained, generates groups where a single PI hires as many PhDs and PostDocs as possible, who do the actual research described in the funding proposals, and publish papers on which the PI is invariably a co-author. This increases the research quality of the PI, resulting in more funding, resulting in more PhDs and PostDocs, resulting in more papers... you get the picture. There is no incentive in this mechanism not to have a big group of low-paid eager employees led by a single PI. It simply works.
So what’s wrong with this picture? A few things, I think. While this mechanism may function, it does not necessarily generate the best possible research for the money spend, and may thus not be in the best interest of the country and the government. Why not? First of all, both PhDs and PostDocs are supposed to be in training by the PI. PhDs more so than PostDocs, but even they are supposed to receive training as a PostDoc is supposed to be a preparatory stage before becoming a lecturer. It’s not hard to see that the more spiky the employment structure, i.e. the more PhDs/PostDocs a PI has, the lower the quality of this training, and thus the lower the quality of the resulting research. Spiky structures also mean low career prospects, and this has a huge demoralising effect on the very people expected to conduct the actual research. Yet there is no incentive in the current monetarising mechanism to limit the number of PhDs/Postdocs for each PI, in fact, the opposite is encouraged.
Secondly, it is reasonable to assume that the best researchers end up being PIs, eventually. Yet in the current system, all their time is consumed in writing grant proposals, and managing their research groups. This is a waste of their potential: they are supposed to be the best researchers available to the community yet they cannot perform any actual bench work! One argument often heard against this is that they at least still generate the ideas that form the basis for grant proposals. Yet it is a well known fact that proposals do not consist of entirely novel ideas. Because of the harsh reviews governments and research councils put in place these days, PIs promise to research things that have already been attained, or are at least within short reach. So many research proposals are not stellar new proposals for groundbreaking research, but bland continuations of existing research, designed to have easily evaluated performance measures to suit the bureaucratic reviews.
And that forms also the basis for the third issue of spiky research groups. While PostDocs are supposedly independent researchers, mentored by their professor, in reality they are performing the research set out by their PI and formulated in part by the grant proposals. This means that they are not fully able to pursue their own ideas, which can be truly paradigm shifting, and again a lot of potential research excellence is lost.
So, how can we go about changing this in a structured manner, that is, by changing the actual mechanism of monetarising academic research? I see two things that can be done by the scientific community. First of all, universities can set a limit to the number of PhDs/PostDocs that a PI can have, based on their obligation to properly supervise. This would ensure that PhDs and PostDocs are properly trained, and that would in turn improve their research output. Universities may be afraid of this because it would limit their star funding attractors, but this is not an objection if they would increase the ratio of PIs to PostDocs, which is the principal problem we are addressing anyway, a problem which I hope the universities acknowledge exists.
Secondly, universities and funding bodies should make it clear that it is ‘not done’ to be a co-author on a paper merely on the merit of having procured the funding for a subset of the authors. Some funding bodies in other countries (e.g. Germany) already make this explicit in their funding rules, and increasingly prominent journals and conferences ask the authors to declare that all authors have made a significant contribution to the research described in the paper (not the research funding). If these guidelines were upheld, they would force PIs to spend more time on actual research for fear of losing their measured research quality, which would mean they would have less time to manage huge groups and write loads of funding proposals, which in turns makes place for more PIs with better structured (i.e. less spiky) groups.
One obstacle remains: recently there has been a push by funding bodies to reduce the number of grant proposals coming in, as a cost cutting measure. The proposals I made above will both lead to an increase of proposals, at least there will be more proposals by more people. This is a good thing though, and the fallacy of reducing funding requests must be tackled. This is one thing where Willetts can act: provide a larger percentage of the budget to the research councils for dealing with grant proposals.
The other way in which Rt. Hon. Willetts can influence this process is by influencing universities and research councils that a de-spiking of the research community structure is a good thing, and that the ideas outlined above should be supported.
FERA2011 Challenge
07/04/11 12:28 Filed in: facial expression recognition | challenges
On the 25th of March 2011, the first Facial Expression Recognition and Analysis challenge (FERA2011) was held in Santa Barbara, California. The aim of the challenge was to hold a competition where all participants competed on a level playing field, with a clear, predetermined protocol. It is described in detail in a paper published in the proceedings of FG’11. Two sub-challenges were defined: the first was in the detection of Facial Action Units (AUs), which encode atomic facial muscle actions in terms of (groups of) muscle contractions in the face. The second sub-challenge was in a more popular yet more contested area: the detection of expressions of discrete emotions.
The data used was the GEMEP-FERA dataset, a subset of the GEMEP database specially created for this challenge. Although the GEMEP database is still not publicly available, the GEMEP-FERA dataset with the AU activation labels and emotion labels, is now available for any scientist wishing to benchmark their AU detection and/or emotional expression recognition system. The data is considered challenging mainly for two reasons: firstly, there is a lot of speech during the recordings, which makes the detection of lower-face AUs a lot harder. Secondly, there is a large amount of head movements. As the challenge results showed, those who attained high competition rankings also had robust methods to deal with this head motion.
The AU-detection sub-challenge was done on a frame-by-frame basis. This was possible and valuable because in the duration of a single video, multiple AUs are activated at different times. It is also not uncommon for the same AU to be activated more than once during the same video. The emotional expression recognition sub-challenge was done on event basis, with each video given a single, discrete label corresponding to an emotional description. This was necessary, because the videos were cut such that they would start immediately with the expression, and ended still in that expression.
The challenge was a great success, with over 90 attendants to the workshop in Santa Barbara (even though it was the first day with decent weather, and held after the main conference). In total there were 15 participants, and 11 accepted papers. The person-specific emotional expression recognition problem appeared to be solved: with the top-three ranking participants attaining 94%, 96%, and 100%. The AU detection problem appeared to be much more difficult, and it seems that there is still a long way to go before this problem is solved. Only 5 participants contributed to this sub-challenge, and the highest scoring team attained an F1-measure of 62% .
The presentation that I gave with a meta-analysis of the challenge and the competition results is now available. The results can also be found on the FERA2011 website.
The data used was the GEMEP-FERA dataset, a subset of the GEMEP database specially created for this challenge. Although the GEMEP database is still not publicly available, the GEMEP-FERA dataset with the AU activation labels and emotion labels, is now available for any scientist wishing to benchmark their AU detection and/or emotional expression recognition system. The data is considered challenging mainly for two reasons: firstly, there is a lot of speech during the recordings, which makes the detection of lower-face AUs a lot harder. Secondly, there is a large amount of head movements. As the challenge results showed, those who attained high competition rankings also had robust methods to deal with this head motion.
The AU-detection sub-challenge was done on a frame-by-frame basis. This was possible and valuable because in the duration of a single video, multiple AUs are activated at different times. It is also not uncommon for the same AU to be activated more than once during the same video. The emotional expression recognition sub-challenge was done on event basis, with each video given a single, discrete label corresponding to an emotional description. This was necessary, because the videos were cut such that they would start immediately with the expression, and ended still in that expression.
The challenge was a great success, with over 90 attendants to the workshop in Santa Barbara (even though it was the first day with decent weather, and held after the main conference). In total there were 15 participants, and 11 accepted papers. The person-specific emotional expression recognition problem appeared to be solved: with the top-three ranking participants attaining 94%, 96%, and 100%. The AU detection problem appeared to be much more difficult, and it seems that there is still a long way to go before this problem is solved. Only 5 participants contributed to this sub-challenge, and the highest scoring team attained an F1-measure of 62% .
The presentation that I gave with a meta-analysis of the challenge and the competition results is now available. The results can also be found on the FERA2011 website.
OpenCV and large AVI files
28/02/11 16:46 Filed in: OpenCV | Programming
Today I tried to let the LBP-Based Action Unit Detector LAUD loose on some of the SEMAINE recordings. The first hurdle to navigate was, of course, what codecs OpenCV can handle. Now, this depends on how you compile OpenCV, and if you compile it with ffmpeg support, which in turn is compiled with e.g. H.264 support, there’s a lot of codecs OpenCV should be able to handle. For those who don’t want to go into this, for whatever reason, there’s always the option to use a codec that OpenCV can always handle.
There is a little information on this on the OpenCV wiki. I thus converted some interesting SEMAINE recordings using mencoder:
There’s a catch here though. Using a raw output codec means that you’ll end up with incredibly big files (around 8 GB in my case). The AVI format lists data chunks in a RIFF table, which apparently has a fixed length. If you exceed this length (because you have too much data), a new table will be started. That’s fine. The problem is that OpenCV does not recognise this, and once the first RIFF table is done, it grabs an empty frame. Usually this means your program exits gracefully (assuming you’ve done the right thing and checked whether a frame was grabbed).
The solution to this? As far as I’m aware there is none so far. But if I find one, I’ll keep you posted.
There is a little information on this on the OpenCV wiki. I thus converted some interesting SEMAINE recordings using mencoder:
$ mencoder in.avi -ovc raw -vf format=i420 -o out.aviThere’s a catch here though. Using a raw output codec means that you’ll end up with incredibly big files (around 8 GB in my case). The AVI format lists data chunks in a RIFF table, which apparently has a fixed length. If you exceed this length (because you have too much data), a new table will be started. That’s fine. The problem is that OpenCV does not recognise this, and once the first RIFF table is done, it grabs an empty frame. Usually this means your program exits gracefully (assuming you’ve done the right thing and checked whether a frame was grabbed).
The solution to this? As far as I’m aware there is none so far. But if I find one, I’ll keep you posted.
Visiting David Tax
17/05/10 19:02
Visited David Tax again to work on our collaborative AU analysis projects. We had plenty of good ideas. It always strikes me how much you can get done with a little bit of face to face time. We had a good discussion about the machine-learning aspect of the temporal analysis of AUs. At the moment our group is the only one working on this, but shortly we will announce the release date of our onset-apex-offset annotation on posed data. We hope this will stimulate research on this aspect of facial expression recognition.