Doing a systematic review  2

One of my papers is going to be a systematic review, so for the past few weeks I have been busy searching and selecting articles. Of course I encountered some problems, because it would not be a PhD if everything went right the first time! Saskia thought it might be interesting if I share some of my experiences with you guys.

 

Search terms

So I don’t know about you guys, but the subject of my research has many synonyms. While I usually refer to my topic as being “intermediate assessment” other researchers have talked about frequent testing, continuous testing, or I don’t know how many other combinations. The problem with topics that have a lot of synonyms is that you can miss some very relevant articles, just because, for example, you only searched for testing and not for exams.

To avoid this I re-read some articles I have used in my research before to see what terms they used. Another smart strategy to define your search terms is looking for other reviews about the same subject and see which search terms they use. Of course if you do find other reviews, make sure that yours is an interesting addition!

After going through some articles I read before and thinking of common synonyms myself I decided on a list of six words that could all signify assessment and seven terms that could indicate some form of intermediateness.

 

Digital illustration of magnifying glass

Searching

So you’ve decided on search terms, now it is time to search! There are so many different search engines and I do not pretend to know them all, but I have used the ones that are most common in the field of educational science, which are ERIC, PsycINFO and Web  of Science. Your professor or librarian may know which are most suitable for your field.

Fun fact. When I asked Web of Science to search for a combination of my 13 search terms, it found almost FIVE.HUNDRED.THOUSAND. results. Even after refining by saying I only wanted articles related to education and educational research, there were still over 8800 relevant articles. As you can imagine this is too much to go through, so I decided to switch to search term pairs. This means I had to do several searches and for now I have only combined my seven adjectives with “assessment”. In the future I may expand my search into the other synonyms for assessment, but for now I’ve decided to go and see how many articles I can select from this.

 

 

Selecting

After running a search you are left with a long list of articles that you need to check for relevance. To make efficient use of your time this first check for selection is usually done based on the title and abstract of an article. Of course this is where you run into problems if titles and abstracts are not informative enough for you to make an informed decision. For these doubtful cases you may need to read the full article.

Furthermore, selecting is a human task, meaning that it is not totally objective. I supplied my daily supervisor with one list of search results and my criteria for selection to check my process. Whereas I selected 33 articles, he only selected 24 and of course he selected some articles that I didn’t and vice versa. So now we need to sit together and see why we chose such different articles.

If you’ve selected articles you need to download the full text for reading. Fortunately, I found 260 full text articles, out of 281 selected search results.

 

required-reading

 

 

 

 

 

 

 

 

 

Reading

After selecting the articles you can start reading and extracting the relevant information for your review. This is what I am starting to do today, so I cannot provide any tips or experiences yet, but maybe you guys have some top tips for me? For example how I am going to survive reading 260 articles in the coming months!!

PhD: Prepare for humdrum Drag?  0

Although many of my friends keep asking me “How do you do this [PhD]?!” (and I keep thinking: Doing what? I just started and I ‘still’ like it…), this post won’t be about self-discipline, self-management, self-reflection, multi-tasking, meta-cognition or another kind of a PhD students’ conditional characteristic.* I enjoyed reading my colleague’s blog on being a PhD student (http://researchblog.iclon.nl/planning-everything-reading-writing-still/) and I could probably write another ten pages about this topic. However, what I chose to share with you today are my experiences with working in an inter-university research project so far.

It has been only 4,5 months since I started my PhD at ICLON. My PhD track is, just like many other PhD tracks, comparable to a (lonely) life on a private island. However, what makes my project less lonely (or less independent), are the people involved in this project; stakeholders so to say. As your research develops, more and more people will get involved. Probably every PhD student will get to the point of receiving feedback from various people with different perspectives, different goals in mind, or at least different pathways leading towards a certain goal. The starting point of my research project is: 3 universities, 3 interdependent projects, and 9 interdependent researchers. Yes: N-i-n-e. I thought it would be nice to tell you a little bit about my who-what-were-when-dilemmas a.k.a. Prepare for humdrum Drag (PhD). Note (!!!): This is not a manual including Do’s and Don’ts but simply a description of my thoughts. It also isn’t a genuine portrayal of reality but rather an exaggerated sarcastic one.

 

  • WHO’s the lucky one? Collaboration. I find it an intriguing phenomena as its process is dependent upon so many different factors. Among others: Expectation management. The projects we’re working on is interdependent in the sense that we’re dependent on each other’s data. This is a good thing, from a sharing-interests-perspective, as a smooth collection of data is close to everyone’s heart. However, from a responsibility-perspective it’s a little bit more complicated. Who’s doing what, where, and when? In other words: Who feels responsible? Me. For everything. Duh, that’s why I’m a PhD student (did somebody say control freak?).

 

are-you-a-control-freak

 

  • WHAT are you talking about? Something can make perfect sense to me and make no sense to others at the same time. It has only been 4,5 months since I started and I already get the feeling of being a puzzled researcher in a world of my own. What I mean to say is; Research is a process in one’s head. Before concrete products appear (such as a theoretical framework, concrete measurement instruments, or even black on white results), we think, read, rethink, reread, etcetera. As a result, meetings so far exist of brain-storming, weighting pros and cons, reformulating ideas, etcetera. In other words, the challenge is to translate our personal thinking process of, let’s say four weeks, to a meeting of two hours (with 8 other researchers also having their own thoughts).

 

justin-bieber-what-do-you-mean

I finally got to understand Bieber

 

  • WHERE?  A picture can say a thousand words…

 

1

Where to save files & notes?  

 

2

Where to synchronize latest versions to?

 

  • WHEN: To mail or not to mail? I think this type of humdrum drag is comparable to the Dutch saying “He shot all his gunpowder” (Hij heeft al zijn kruit verschoten). On a normal working day several issues will cross your mind that you would like to discuss with others. Will you decide to wait and think about it yourself for a couple of hours or days more? Or will you mail your colleagues and ask for their input? In the latter case, be aware that 1) This may result in a day of mailing back and forth without actually moving forward (my personal record is 21 mails of 7 people within 2 hours on 1 specific topic), or 2) Your colleagues will be fed up with your e-mails and won’t respond to your e-mails a second time.

 

* I realize that, in the end, I actually did write about it…

 

 

 

 

 

 

A Pirate’s Treasure Trove  0

Last time, I was thrilled to announce that my data had arrived. Now, as I am looking though the piles and piles of questionnaires, new questions have risen in my mind:

  • Why would someone cut out squares from the pages of their questionnaire?
  • I said: “tick the boxes in black or blue pen”. Not: “tick them in red, orange, yellow, lightblue or barely visible pencil”!
  • Why do these kids leave so many questions unanswered? I didn’t put that question mark in for nothing!
  • Why does a significant portion of the student population think my picture on the announcement letter ought to have a beard?

But the most important thing I discovered was that not all students fill in the questionnaire seriously. Of course, that could have been expected, but naïve as I was, I thought  it wouldn’t be so bad. While scanning in the hardcopy questionnaires into the computer that can read the answers for me, I came across a group of 6 questionnaires that were all filled out in a zigzag pattern. Obviously, this had been a group of silly teenagers who were sitting together and had come up with the idea of transforming the questionnaire into a zigzag artwork. Unfortunately, I could not use these questionnaires anymore. From then on, before I scanned the questionnaires, I manually looked through all of them to discover any other potential jokers. While doing this, I also discovered some other nice (and some not so nice, and some even very rude) artworks of students on the questionnaires. I made a selection of the nice ones as a keepsake.20151217_143010

“Good luck with your research, stranger”. Well thank you, stranger!

20151218_151800

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Nice hat!

20160122_103338

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“This makes me cranky”

Collage

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

An assembly of some nice artworks.

20151119_163555

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

A Harry Potter-with-a-beard look and a mathematical problem.

20150917_153019

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Some students thought I should be a pirate instead of a researcher. Time for a career swap?

 

It is a little like snail mail: students responded to my “letter”. Although not very extensive or polite sometimes, it was a peculiar and surprising form of communication. I have enjoyed looking for the most creative outbursts on the paper.

Which one is your favourite? And have you ever experienced students drawing on your questionnaire or picture – and if so, did they also think you were meant to be a pirate?

Until next time mateys, ARRR!

Pirate treasure

 

 

 

 

 

 

 

 

 

 

 

 

Well, this pirate is going to search for more treasures in her data!

Image source: blogs.disney.com

My Questionnaire Quest  0

Quest: a journey towards a goal, serves as a plot device and (frequently) as a symbol… In literature, the objects of quests require great exertion on the part of the hero, and the overcoming of many obstacles”.

From last December until now, I have been anxiously awaiting for mail in my mailbox here at ICLON. By the time it was Christmas, I was singing “All I want for Christmas, is maiiiiil”! Really, nothing could make me a happier PhD student than receiving a bunk of mail like this on my desk every week:

20151216_090120

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Because, ladies and gentlemen, my first data have arrived! Finally tangible proof that my first year of work as a PhD student has paid off.

Constructing and distributing a questionnaire was somewhat more complicated than I thought. Here are some things you should think about:

 

  1. Constructing the questions. You cannot just ask whatever you fancy to ask, I discovered. You need to justify what you ask your participants, preferably supported by literature and/or by questionnaires already available covering your topic.
  2. Online or hardcopy’s by mail? That was a very relevant question, since I wanted to have about 2000 students filling in my questionnaire, an online survey would be so much easier for me. But it would also have such a small response rate… Also considering that my questionnaire took about 20 minutes to complete, I chose to send around my questionnaire in hardcopy. That way, students could fill it out in the lesson of the corresponding teacher, and be more likely to complete the survey.
  3. Recruit participants. In my case, I needed schools and teachers to commit to my research, ensuring their students and teachers would fill in my questionnaires. I already knew some teachers, but definitely not enough to cover my whole research population. So, I started asking around. With colleagues (do they have any contacts with schools that might be interested?), friends, acquaintances, institutions also interested in my research topic… And eventually, I even made a list of appropriate schools, looked up their telephone numbers and tried to call the specific teachers to explain them about my research and invite them to participate. Do not underestimate this step. It. Takes. Time.
  4. Logistics. After I printed and stapled over 2000 questionnaires (thank God for automatic staplers), they needed to get to the right persons in the right schools. And those right persons in the right schools should also be able to send the piles of paper back to the right person: me! I am so, so lucky to have gotten help from people in my department, and people from the post office (and occasionally my boyfriend, who helped in the stapling process). I had this whole administration of how many questionnaires should go to which school, how many for students (white coloured paper) and how many to teachers (orange coloured paper). I needed piles of envelopes with the right addresses, and also self-addressed envelopes in which the teachers could send the questionnaires back in the mail.
  5. Communication. It helps to be clear about the agreement you have with the corresponding teacher. Remind them in which classes the questionnaire has to be distributed. In general, remind them. And administer who returned how many questionnaires to you.

 

And then, everything needed to get back to me. I waited in such anxiety. At this moment, my response rates luckily seem very high, although one of my greatest fears also came true as some of the envelopes got lost in the mail. While still awaiting the very last envelopes to return, the scanning and analyses can begin…

What are your experiences when constructing and distributing a questionnaire? What were the obstacles you met, and do you have any tips and tricks for others? Please let me know in the comments below!

cartoon8

 

 

 

 

 

 

 

 

 

 

 

 

Image source: www.channel4learning.com

Killing two birds with one stone: Audit trail to secure research transparency and accountability  0

 

Until recently, academic integrity within Dutch educational research was largely a matter of individual and collective responsibility, rather than applying specific compliance measures. University researchers are expected to conduct themselves in an ethical manner with respect to the ways in which they design, carry out and report academic research. However, procedures and requirements with respect to ethical conduct and academic integrity practices of research projects are changing. Research financers and scientific journals require more strict procedures, incidents with research integrity, safety and privacy call for more transparency and accountability, and recent developments in the domain of open resources trigger questions about accessibility and re-use of data.

 

Much of the literature is framed in terms of misconduct or academic corruption with research ethics and tends to focus on the negative framing of academic integrity as “corrupt” or “bad” practice. Fabrication and falsification of results together with plagiarism and ethical abuses practiced by academic researchers are frequently highlighted. Yet ICLON Leiden University Graduate School of Teaching designed a procedure that builds on trusting academic researchers to conduct themselves in an ethical manner with respect to the ways in which they design, carry out and report academic research.

 

A distinctive character of the ICLON research program is its double focus of developing educational theory and practice, which means that the research projects of the program aim at simultaneously contributing to the improvement of educational practice and generating knowledge about this practice. Because of this double focus, many research projects of the research program are highly-contextualized and are characterized by complex research processes which ask for many interpretations of the researchers and lack standardized procedures of analysis.Kill 2 birds 3

 

Akkerman and colleagues designed and evaluated a so-called audit procedure to ascertain if this kind of studies meets the criterion of trustworthiness. This procedure is about the visibility, comprehensibility, and acceptability of the research process. A decision in the research process must be made explicit and communicated to be judged at all and substantiated to be judged by its logic and content.

The data management procedure of ICLON builds on this audit procedure and includes an audit trail, which allows an auditor to track, understand and assess the research process from the final conclusions as reported in a paper back to the data. This data management procedure is considered a practical and useful way to secure both transparency of the research process and accountability of its researchers. It can also be understood as a way to support researchers to improve the quality of the research process and to raise their awareness of the importance of academic integrity.

So, what did I learn from my six-week stay abroad?  0

As you all might have noticed, we went offline for a little while. As much fun as we have when blogging about our PhD-lives, we need ideas for interesting blogposts. So, we took a little break, allowing ourselves to have experiences we could later write about. And look, here we are, back with new blogposts :-).

 

This week, I am taking you with me in a flashback to my stay in Berkeley. The previous time I wrote, it was mainly about my preparations for and my first experiences in Berkeley. Now, I have been back for a little over two months, and will write about what useful knowledge and experiences I gained. Because even though I was there for ‘just’ six weeks, what I brought back will last me months! In this blogpost I will focus on two of those insights I’ve gained. Read more

Letting off STEAM  2

071506nanotechnology02.ai
Image source: www.ontariosciencecentre.ca

 

As an educational researcher, I focus on the bèta science subjects in secondary education. Internationally, these subjects are also often referred to as STEM: Science, Technology, Engineering and Mathematics. Some time ago, I heard the suggestion to add a letter to this formula: the A of Art.

 

This is food for thought. In my research, I tried to briefly determine what people thought that an engineer, or a designer needed. Among other characteristics, I heard that a lot of people associated these professions with being creative, and being able to build things. Also, when designing, you often go back to your previous steps and adjust your choices to optimize the result. Some people even mentioned they thought of a designer (or engineer) as an artist. These qualities are not only used by designers and engineers, but often also by artists. Qualities that are considered to be important in STEM, appear to be important in the arts as well (or vice versa!).

 

I think Arts and Sciences are a good match. For one, you can communicate science through art. I remember a conversation I recently had with my peers, about valorisation of our research and letting people know your results. Getting into the headlines of newspapers is hard. Why not use art to communicate scientific research? Art can be a great way of enhancing one’s experience with science, not only addressing their ratio, but also their emotions end feeling for aesthetics. And why not use art in science? A creative infographic could really light up your paper. Art is also valuable in science education: as a trained Biologist, I drew hundreds of plants, animals, cell structures… Drawing enhances the amount of detail in which you look at things, which ultimately leads to a greater understanding of that what you’re studying. Art, in terms of music, could help you better understand the scientific content of a message: for example in this Dutch DNA song I made with peers during a Science Communication course.

 

steam
Image source: adapted from www.studiostift.nl

Arts and sciences are combined during festivals, such as the Night of Art and Knowledge, which took place last month here in Leiden. The several activities present included music acts (with attention to the science of music), maths in comedy, arts inspired by astrophysics, science themed theatre and storytelling, and more. The next Night of Arts and Sciences is already in the planning. The theme of this year’s children’s book week in the Netherlands is related to science. I often consider children’s books also small pieces of art. They do not only communicate knowledge, they communicate feeling as well.

 

In my opinion, arts and sciences can enhance each other when combined. Please comment below if you know more ways in which they might enhance (or possibly hinder?) each other. Do you think there is a future for STEAM? Feel free to share your thoughts (and let off steam).wink-98461_640

 

Ruimte voor excellentie?  0

Iedere professional heeft ruimte nodig om goed te kunnen functioneren, of niet?

Van sommige professionals wordt zelfs verwacht dat zij deze ruimte zelf creëren en benutten. De beginnende docenten in het Eerst-de-Klas traject  en het OnderwijsTraineeship  zijn hiervan een voorbeeld. De overheid heeft deze speciale leerwerktrajecten van de lerarenopleiding opgezet om excellente academici te enthousiasmeren voor het onderwijs. In het voortgezet onderwijs zouden deze academici een optimale ruimte dienen te creëren en benutten om hun baan uitdagend te houden voor zichzelf en een innovatieve wind te laten waaien in de school. Mooie woorden worden gebruikt in de omschrijving van deze programma’s. Zo is er bij het Eerst De Klas traineeship een leiderschapsprogramma ‘dat is vormgegeven door de meest toonaangevende organisaties in Nederland.’ Het OnderwijsTraineeship biedt naast de lerarenopleiding ‘masterclasses die je internsief laten kennis maken met de volledige breedte van het onderwijsveld’.

Toch lijkt er een belangrijk mechanisme over het hoofd te worden gezien. De docenten in beide trajecten starten namelijk als beginners in schoolorganisaties en hebben tijd nodig om de organisatie te leren kennen en de eigen positie binnen de ‘gevestigde orde’ te zien. Een begrip als enculturatie past goed bij deze fase. De balans tussen ruimte creëren, benutten en krijgen is daarmee een belangrijk onderdeel van deze initiatieven. Wij (Jacobiene Meirink & Anna van der Want) zijn heel benieuwd naar hoe de huidige docenten in het EDK en Onderwijstraineeship de balans tussen creëren, benutten en krijgen van ruimte ervaren! In een door ProBO gesubsidieerd project onderzoeken we deze thematiek.

Voor meer informatie zie www.professioneleruimte.info

 

A PhD-student abroad: My first experiences from across the ocean  1

So, last Friday two weeks ago I left the gray-skied Netherlands for my big adventure to the United States. I was going to visit UC Berkeley for six whole weeks! I would be able to meet faculty from UC Berkeley and talk to them about my and their research. Before my departure I was, of course, nervous:

  1. I had a wish-list made for what to get for my research from this trip, would I be able to cross off everything on it?
  2. I would have to talk to many new people, every time explaining my research all over again and have it open for discussion

In this blogpost I want to share some of my experiences with you regarding these two points I just made.

 

My wish-list

It all started for real last Monday. I had my first meeting with my contact at UC Berkeley. In the 45 minutes I was in her office things went really fast; names of people I should really meet flew by, we discussed what I wanted to get from my stay at UC Berkeley, and I was suggested a book to read. After this meeting I soon realized that I probably won’t have to worry about checking off everything on my wish-list; it will probably get done. For, all the people whose names were mentioned during the meeting, would be told of my visit at UC Berkeley, which gave me an opening of contacting these people myself. Of course, this still gave me the, sometimes difficult, task of introducing myself a little over email, thinking of the reason why I would meet that specific person, and asking whether the person would have time to meet with me. Luckily, up until now everybody has responded positively. In addition, I have also visited some classes in the teacher education programs. And I am visiting some research group meetings. Thus, even after only two weeks, I can honestly say much on my wish-list has already been crossed off. I can start looking for some extra wishes :-).

I think what really helped in getting so far with my wish-list, is that I communicated it and I keep communicating (parts of) it to the people I meet with. For, they are the ones who can help me getting everything done. And although I now use this strategy on a visit abroad, I think this is very helpful in many different situations, also. I mean, if there is anything you want to do during your PhD, communicate it with the people around you. It is good if you want to find stuff out for yourself. I’m not saying you shouldn’t do that. But you will notice that with some help from others, you might get a little further, maybe the person you’ve just communicated your wishes to, knows just that specific person who can help you in achieving your goal.

 

Meeting new people everyday

The fact that I am crossing wishes off my wish-list, means that I have to meet new people everyday. That also means that over and over again, I have to explain my research. You would say that if I have to do that so often, it becomes easier and I can just hit the replay button that makes my brain tell my mouth to talk about my research. If only. It means that every time again, I look critically at what I am explaining and to whom. First of all, for almost everyone around here, I have to tell more about the context in which my research is taking place, than I would have to in the Netherlands; not everybody is familiar with the Dutch educational system. Second, you have to think about who you have in front of you and what his/her research interests are. I have experienced that the meeting will follow a much more natural course, if you find some common ground to talk about. For my meetings up to now, I didn’t have a whole set of questions prepared. I did some research about the person, came up with two, for me important, questions and went to see the person. Sometimes, I haven’t even used the question I had made up before, but it was good I had them. We would just find concepts to talk about that have kept both our interests and the meeting would just go. Still, I find it every time a challenge to think about how I am going to talk about my research with the next person, and what questions I want to ask him/her. So far, however, it has really paid off; at the moment I am really thinking over some concepts and my theoretical framework. But, the story behind that will follow in a later blogpost.

 

Well, this blogpost has been a lot about my experiences, but what message do I want to bring to you with this?

  1. Make a wish-list (about what you want to achieve: during your time as a Phd-student; at a conference; when visiting another university, etc.) and communicate it
  2. Look up people you are meeting and think about how you want to tell your story to them
  3. Think beforehand of some questions to ask the person you are meeting
  4. And, if you are going abroad, don’t forget to have fun too 😉

PhD Side-quests: Teaching  0

Tweet Tim

A while ago, my co-worker and fellow blogger Tim posted a tweed likening getting you PhD to a MMORPG. I thought this was a funny and also pretty accurate description and today I want to write about one of my favourite PhD side-quests: Teaching.

 

For context, I write this as a full time PhD student who is not expected to teach that much. About 10% of your appointment is a normal amount of teaching for PhD students in the Netherlands. This may be very different from the situation of some of you. Some of you may be teachers/PhD candidates whose main quest is actually teaching.  If it is, my post may not be that useful for you, but hopefully you have some nice tips to add for those of us who are inexperienced teachers.

 

Before I started my PhD I worked as a Teaching Assistant in various courses for two and a half years and I’ve been involved in the teaching of two different courses since I started my PhD. I love teaching and think it is very rewarding, but it is also very tiring and time-consuming. Today I share some tips to make your teaching experience as good as possible:

 

Educate yourself

If you’re a new PhD student teaching a course or tutorial may seem intimidating. Especially if you have never taught before! There are several ways to educate yourself.

You can try and see if your university offers workshops or courses for teachers to improve their general teaching methods. You can also talk to more experienced co-workers and see how they approach their teaching. This can be in general, or related to the specific course you’re expected to teach. Experienced teachers hopefully will have hands on information that you can use while preparing for teaching.

 

Be well prepared

A thorough preparation is vital, especially if you expect that you will be teaching a course several times. Of course preparation is time consuming, but it is a timesaver if you are able to take your slides from previous semester. A thorough preparation can also help with nerves on your side. If you are really well versed in the subject you will be teaching, you will feel more secure standing in front of the students.

 

Plan!

Teaching can be a serious time consumer. Especially if you have to grade student work and provide individualised feedback. One tip that a co-worker gave me is to set aside a specific moment during the week for these type of activities.  That way you know how much work you actually have to do and you have some control over the amount of time you spend. Try to be realistic in your planning. I for one know that I cannot go straight from teaching to being focused on my research, so I’ll plan for some transition time as well.

 

What’s in it for you(r research)?

When your teaching duties consist of thesis supervision it may be possible for you to combine teaching and your own research. Often, the students will get involved in your project and (hopefully) collect some of your data. But even when your teaching is not that directly linked to your research there may still be links. Maybe the content of the course you’re teaching is closely related and you can use your theoretical insights during instruction. But even if there is no way to connect your teaching to your research, teaching a course is relevant to your personal and professional development.

 

Enjoy it

Whatever you do, enjoy your teaching experience!

Happy questing and hopefully this side-quest will help you level up!

wink-98461_640

 

Have you been teaching and what are your tips for PhD students who are starting to teach?

 


Translate »