Marking of papers

[BedfordBoy]

One of my colleagues who will be sitting the exam in the UK, but who has obtained almost all his experience abroad, has got me wondering about the exam marking process - specifically who marks the papers and how do they maintain consistency between markers?

Also, are all the papers sent to London for marking?
If they are, how are candidates marked who have answered the papers based on their own local signalling principles and equipment?

If they are not, how does the IRSE make sure it isn't easier to pass in one country than in another?

Does anyone here have any insight as to the process, or any other thoughts?

[PJW]

One of my colleagues who will be sitting the exam in the UK, but who has obtained almost all his experience abroad, has got me wondering about the exam marking process - specifically who marks the papers and how do they maintain consistency between markers?

Also, are all the papers sent to London for marking?
If they are, how are candidates marked who have answered the papers based on their own local signalling principles and equipment?

If they are not, how does the IRSE make sure it isn't easier to pass in one country than in another?

Does anyone here have any insight as to the process, or any other thoughts?

Examiners know if a student is sitting the exam abroad from the style of candidate number; your colleage MUST make it very clear on ALL their answers which practice they are familiar with- the default for someone who has failed to specify but is sitting in the UK would obviously be UK. This would be wrong in the case of your colleague- but would be their fault as all candidates are always advised to state!

This is what I believe from knowing many of the examiners, most of whom are from the UK, generally engineers in their 30s/40s/50s/60s. Papers are marked independently by 2 examiners; often there are only 2 examiners per module so both mark all papers; in other cases 3 examiners are used, of which 2 mark any one paper but in all combinations. The separate marks are compared and if the marks differ wildly or close to a grade boundary are discussed and re-evaluated as necessary. I believe that the whole exam committee then meet to discuss overall perormance in the various modules. It is IRSE Council that authorise the publication of the results, generally at a committee meeting mid December.

I believe that all papers are sent to London and then the copies sent out to the various examiners. Actually a candidate from abroad in my opinion may have a higher chance of "getting away with something" than a domestic candidate. If what they declare as their practice sounds feasible and self consistent then it probably won't be challenged; if it seems odd then the onus is on the examiners to take advice from some member off the IRSE or other appropriate person who does have knowledge that could confirm that the suspect answer is actually correct for that environment.

Nothing is perfect but I believe that the marking is as fair as it can be; I would actually have greater concerns about the questions. I know that the questions are now reviewed by a non native speaker and also with special consideration from those with a non UK background. However I'd find it hard to assert that there is no unconscious bias, but I do think that the IRSE does what is reasonable, given the current membership of the Institution and the students puttting themselves forward for examination.

An example where it is really impracticable to improve but things are not ideal is that module 3 Control Tables are always on a layout that is definitely very UK Mainline (and actually often very 1970s/1980s)- it must be hard if answering for an environment that is very different and based on different assumptions- a candidates local practice may not fit the situation at all, as that form of track layout just wouldn't exist. Having said that, many candidates ask why the IRSE won't prescribe one set of standards and one design of Control Table- you can't have it both ways!

The reality is that all those that sit the exam are English speakers even if not all have English as their first language. Most also either work abroad for the UK railway or work on a railway where UK has had historically a significant influence. Although the IRSE strives to get "foreign" papers at meetings, conferences, conventions and IRSE News and I feel actually does very well at this, there is inevitably a "home" bias just by weight of membership. I deliberately tried to get international input for module 2 Study Pack- got some useful input from Australia but various attempts to get something from the Netherlands (close neighbour, some great similarities some significant differences, widespread English) for example proved unsuccessful.

Hope your colleague passes and next year will be able to contribute their experience to this Forum and perhaps YM events that are in the initial planning phase!

PJW

[BedfordBoy]

Thanks, that's a very comprehensive answer. I told my friend to put down what principles he was applying, however his main concern last week was that past papers asked for controls for call-on & warning routes, and he had no idea what they were.

Still, after I explained them to him the afternoon before the exam he seems quite confident that he passed (apart from the usual problem of running out of time, but that probably deserves another post - why does module 3 test how quickly we can write rather than the extent of our knowledge?).

I think that even though a candidate can technically answer a paper based on the systems they are used to, it seems that module 3 in particular is very much biased towards the UK in asking for C & W routes.

[PJW]

Thanks, that's a very comprehensive answer. I told my friend to put down what principles he was applying, however his main concern last week was that past papers asked for controls for call-on & warning routes, and he had no idea what they were.

Still, after I explained them to him the afternoon before the exam he seems quite confident that he passed (apart from the usual problem of running out of time, but that probably deserves another post - why does module 3 test how quickly we can write rather than the extent of our knowledge?).

I think that even though a candidate can technically answer a paper based on the systems they are used to, it seems that module 3 in particular is very much biased towards the UK in asking for C & W routes.

I'd agree with the last; indeed that was the sort of thing to which I was alluding. However should declare their practices and if this means that C and W routes do not exist then they'd have to declare that and they will just have to be marked for what they do do; given that the available marks would have to be allocated over fewer routes they would therefore be expected to get them more accurate as would have more time. Examiners would try to be "fair" - but obviously hard to be demonstrably so as inevitably rather subjective.

What does concern me that they seem first to have raised last week; one would have assumed that it would have become obvious from the first time they read the syllabus, Study Pack, Past Paper......

All the modules are a speed test; I agree that this is not 100% sattisfactory. However it is for a range of reasons:
a) "tests the system" under pressure
b) being able to quickly get to grips with something is actually something of a mark of real familiarity / experience; it is a way in exam conditions of judging people. I am a tester and in the real world peple that think they can do the job but fiff and faff about are really "not competent", even if they can do what is needed in a training environment,
c) it does strike me as odd that for the signalling profession when "getting a relatively small amount of design absolutely correct" is more important than producing a vast quantity that is nearly right, that the exam does the opposite. The reality is that it would just take too long otherwise and actually it is a damn sight easier to get everything 80% right than getting 50% of the workload 10% right. Examiners are experienced in doing what it is reasonable to do in the time; if you can do it right but only but taking an excessively long time then that does not rate as adequate performance,
d) the exams are long enough at present and splitting over more days would be particular disadvantage for those with long journeys to exam centres, increase costs and effort that has to be expended by volunteers. It is felt important that someone who wishes to do so can do 4 modules in a single day.

However personally I do fundamentally agree with you that there is too much time pressure and too much "exam technique" rather then "signal engineering" in the exams. I felt this when doing myself (was it really a quarter century ago!). I still feel this again now (have helped people with exams since about 2000); I certainly feel the exam is not ideal but it is the best we have at present and it isn't obvious how to change yet retain what is good.

I think that the IRSE is receptive to constructive criticism; indeed I have managed to win a few small battles (doubled the reading time, mark allocation within written questions) etc. I think that if it were my decision I'd make certain changes, but it isn't easy to know the best thing to do; there is a flip side to many suggestions that seem initially a good idea. I am willing to campaign for more changes but only when I am personally convinced and feel that I have a rational argument and realistic proposal to make.

Need to be careful not to criticise without having a constructive alternative to offer. I have felt for some time that the Study Packs could be improved (some I feel noticeably better than others); I have now had the chance to make my attempt at producing one for module 2 which atttempts to be far more of a teaching guide than the others. It is a bit of eperiment to see whether it is actually used, whether it is found helpful and whether exam results improve now that one is available- if they don't, then won't have been worth the not inconsiderable effort. The jury is still out on this one.

Similarly I think some of the difficulty with mod 3 would be ameliorated if there was a "standard" defined that candidates could learn to be able to do the exam, even if it wasn't their everyday practice. In a sense Alex is a perfect example of such a person- he is in a managerial operational role on an automated metro system on the other side of the world, so UK style Control Tables are "foreign" to him in many senses. He has learnt from books and a training course in the UK and sought help by means of this Forum- he has been by far the most active student so I really hope that he passes this year, but certainly he has made considerable progress. I certainly believe that the Study Pack itself has not been particularly useful to him, but that the material that is slowly building up here is proving of more assistance. It'd be even better if more people were prepared to contribute their attempted answers, post information that they had acquired, enter discussions and generally act more as a virtual Study Group- perhaps with this now as a foundation the students for 2009 exams will feel able to be more interactive.......

I think that both Peter (who set up this site and whose brainchild it was) and I share some of the students concerns such as this; however we are also well aware that the reality is that relatively few make significant efforts themselves to achieve something for their own personal development. The YM committee seem keen to offer increased support to students next year and if we can maintain the active involvement of those who have recently passed the exam themselves, they'd be a better basis from which to argue for any changes to be made. However the thing that I think would really force the change is if the IRSE had greater membership of relatively junior people who then wished to take the exam but coming from backgrounds significantly different to the UK.

regards,
PJW