Jump to content
Andrew Chen

Live Chat: Maintaining Rigor When Research Operations Are in Flux 6/15/20 @ 2pm EDT

Recommended Posts

Andrew Chen

Research team leaders Kip Ludwig and Lique Coolen will be live in the Neuronline Community to engage in a discussion and answer questions about challenges and opportunities for enhancing and maintaining scientific rigor during shut-downs and re-opening of research laboratories and publishing in the times of a pandemic. A recent Neuronline article  sets the stage, but the discussion will be broader as all community input is invited and encouraged. This live chat is part of the Foundations of Rigorous Neuroscience Research (FRN) program.

Post a question in this thread during the Live Chat for Kip and Lique to answer, or post ahead of time if you can't make it!

Meet the live chat hosts:

Kip Headshot 4M.jpgKip Ludwig, PhD

Kip Ludwig leads the Ludwig Laboratory at the University of Wisconsin, specializing in neuromodulation, bioelectronic medicine, and electroceutical devices. The lab’s primary focus is to develop minimally invasive, next-generation neuromodulation therapies to hack the nervous system, treat circuit dysfunction, and deliver biomolecules precisely to target areas. Ludwig previously served as the Program Director for Neural Engineering at the NIH. He co-led the Translational Devices Program at the NINDS, led NIH BRAIN Initiative programs to catalyze implantable academic and clinical devices, and led a trans-NIH planning team to develop the S.P.A.R.C. Program - stimulating advances in neuromodulation therapies for organ systems.

Coolen_Lab_Crop_Small.jpg

Lique Coolen, PhD

Lique Coolen is an associate dean and a professor of biological sciences at Kent State University. She previously served as faculty at the University of Mississippi, University of Cincinnati, University of Western Ontario, and University of Michigan. Coolen earned her PhD from the University of Nijmegen in the Netherlands.

 

 

 

We are evaluating this program at every step and rely on your input and feedback to improve! At the end of the live chat, please fill out the survey below.

Share this post


Link to post
Share on other sites
Lique Coolen

Hi, I’m Lique Coolen, I’m here today to answer any questions and engage in a discussion about challenges and opportunities related to the shutting down and now re-opening of our research programs. I’m also very excited to learn from my fellow panelist Dr. Kip Ludwig and from all of you! 

  • Like 4

Share this post


Link to post
Share on other sites
Kip Ludwig

Welcome to our online chat on maintaining rigor while research operations are in flux. I'm Kip Ludwig, an Associate Professor at University of Wisconsin-Madison and a former NIH Program Director who worked on the NIH BRAIN Initiative and NIH SPARC Programs. Scientific rigor requires a lot of forethought and persistence, and we can easily lose focus as we try to make up for lost time as we start returning to lab. However, virtual time is also a great opportunity to put better systems in place within your lab to promote rigor. I'm excited to be here with Lique to share some ideas on how to do just that! 

  • Like 4

Share this post


Link to post
Share on other sites
Lique Coolen

And just to add: I am a Professor with an active funded research program, re-opening our labs as we chat, while also serving as PI of the FRN program for SFN and Associate Dean for faculty research development, postdoctoral studies, and graduate studies at Kent State University.

Share this post


Link to post
Share on other sites
Guest Nikolay

Hello, is there a live webinar to watch, or is this a text-based chat discussion? Thanks!

Share this post


Link to post
Share on other sites
Andrew Chen
2 minutes ago, Guest Nikolay said:

Hello, is there a live webinar to watch, or is this a text-based chat discussion? Thanks!

Hey Nikolay, this will be a text-based chat! Please post any questions you have for Lique and Kip in this thread. Thanks!

Share this post


Link to post
Share on other sites
Andrew Chen

Submitted question:

Quote

Which neuroscience fields do you think will benefit most from the move to digital data management?

 

Share this post


Link to post
Share on other sites
Susan Tappan

Hi Lique! Hi Kip!  I'm wondering if you can share what your lab make-up currently is: What type of guidelines you're following from your university, and community, and your state? How many lab members do you have? Are you all back in the lab? 

 

  • Like 1

Share this post


Link to post
Share on other sites
Guest guest

Oh - that's disappointing - I thought this was a live discussion

Share this post


Link to post
Share on other sites
Andrew Chen
Just now, Guest guest said:

Oh - that's disappointing - I thought this was a live discussion

Hey there! Kip and Lique are online right now and are typing out answers to people's questions as we speak so ask away! 🙂 

Share this post


Link to post
Share on other sites
Guest Guest

Is there anyway to have this automatically refresh with posts? I keep having to refresh my browser to see anything posted. 

Share this post


Link to post
Share on other sites
Kip Ludwig

Clinical neuroscience has been on the forefront of the move to digital data management out of necessity, but is still grappling with some of the practical difficulties in curation/annotation/automated analysis/statistical analysis. Behavioral neuroscience is just beginning to move to digital data management, but will have to look closely at the hard lessons learned from clinical neuroscience.   

 

Share this post


Link to post
Share on other sites
Lique Coolen
Quote

Hi Susan. Thank you for the question. My lab team includes several graduate students, postdoctoral fellows and research technicians. The trainees are all at different stages of their career. We also have undergraduate students in the laboratory. We received the green light to re-open in phases. During the first phase, we need to wear face masks in addition to regular PPE, maintain social distance, clean all surfaces and wash hands constantly. And all work that can be done at home, needs to be done at home. Thus, we are still doing lots from home, but are also starting experiments in the lab.

 

Share this post


Link to post
Share on other sites
Andrew Chen
Just now, Guest Guest said:

Is there anyway to have this automatically refresh with posts? I keep having to refresh my browser to see anything posted. 

If you have an SfN account and log in, you'll be able to change your notification settings here: https://community.sfn.org/index.php?/notifications/ to enable browser notifications and update you with any new posts. Hope this helps! You can always direct message me for further assistance if needed.

Share this post


Link to post
Share on other sites
Guest eline van der beek

Just curious to learn how you maintain science discussion with the team or project based during remote working. Discussions are far from optimal when using the most common web based tools is my experience

Share this post


Link to post
Share on other sites
Guest James Coughlan

Does anyone have pointers to resources that help researchers move away from paper-based records for human subjects experiments to electronic ones? For instance, what are the best practices for putting sensitive (possibly HIPAA-compliant but not necessarily) data on Dropbox or Google Drive, and sharing these data (when appropriate) with colleagues?

Share this post


Link to post
Share on other sites
Guest Nikolay

Any advise for conducting research (specifically data collection) remotely? 

Share this post


Link to post
Share on other sites
Kip Ludwig
7 minutes ago, Susan Tappan said:

Hi Lique! Hi Kip!  I'm wondering if you can share what your lab make-up currently is: What type of guidelines you're following from your university, and community, and your state? How many lab members do you have? Are you all back in the lab? 

 

Response: My lab is in Wisconsin, and is actually in the same building at the University of Wisconsin-Madison Public Hospital. Wisconsin has had a relatively low number of COVID cases, but given our proximity to the hospital we need to be very careful as we reopen. We also have an open lab structure sharing lab space with multiple labs spanning clinical, behavioral neuroscience, and neurotechnology. So we had to coordinate closely reopening to maintain social distancing during Phase I. We have 10 staff/post-docs/graduate students in my lab, and several undergraduates who will be staying virtual until the end of Phase I reopening.  

Share this post


Link to post
Share on other sites
Susan Tappan
1 minute ago, Lique Coolen said:

 

That's great that you're getting back into the lab!  It's an interesting situation to have parts/steps of an experiment that would normally be done by a single person now being possibly a shared task due to staffing limits or health concerns.  Is this something your lab is currently doing?  Are you using protocols or experimental details that are stored on the cloud or otherwise accessible to everyone in the lab?

  • Like 1

Share this post


Link to post
Share on other sites
Mario Gil

I've been trying to figure out how to help facilitate/grant my students remote access to resources (software, server space) in the lab, but that has been difficult. And some work cannot be done from home due a number of issues (e.g., trainee's personal PCs don't have the memory required to run certain programs). Do you have any suggestions for neuroscientists who may not be as tech savvy as others. Are companies, NIH, others, starting to address this issue? Or perhaps there's software/programs available that I'm not aware of? 

  • Like 1

Share this post


Link to post
Share on other sites
Guest P Verghese

We conduct research that involves human participants, and are considering alternatives to in-person data collection. Our studies involve visual, auditory and tactile stimuli and we are considering alternate means to gather data remotely. Are their resources to provide guidance on the extra precautions that need to be taken during remote testing?

Share this post


Link to post
Share on other sites
Lique Coolen
1 minute ago, Guest eline van der beek said:

Just curious to learn how you maintain science discussion with the team or project based during remote working. Discussions are far from optimal when using the most common web based tools is my experience

Hi Eline! Thank you for joining us. I have been using Zoom to meet with each of my lab members individually at least several times per week. We were able to share screens and look and work at data files together. It worked very well for us and I am even going to keep using it while we go back to the lab. Like you, I have many other responsibilities and need to work remotely. This way, my students can meet with me much more often and I can mentor them (even) better.

Share this post


Link to post
Share on other sites
Guest Santani Teng

Some types of data collection can be moved to remote platforms, but not all. I would be curious to hear how folks approach the larger issues of reframing research questions to fit the available methods (eg. instead of a neurophysiological hypothesis, how would [X] process be reflected behaviorally), and how much a funder could/should be expected to accommodate that shift.

Share this post


Link to post
Share on other sites
Guest Laura O'Dell

Thank you for doing this.  Are there materials available to provide for trainees on rigor and reproducibility?  Also, how do you suspect that journals/reviewers will be flexible when we are faced with the limitations in animals, resources, etc.  Again, thank you both for your time!  laura

 

Share this post


Link to post
Share on other sites
Teresa Murray

What platforms do you recommend to monitor and discuss a wide variety of data? For example, my lab uses video and microscopy images, and electrophysiology and biosensor data to study live rodents in disease and injury models. It is cumbersome to review data in these diverse formats on Zoom.

Share this post


Link to post
Share on other sites
Kip Ludwig
3 minutes ago, Susan Tappan said:

That's great that you're getting back into the lab!  It's an interesting situation to have parts/steps of an experiment that would normally be done by a single person now being possibly a shared task due to staffing limits or health concerns.  Is this something your lab is currently doing?  Are you using protocols or experimental details that are stored on the cloud or otherwise accessible to everyone in the lab?

Response here: In Phase I, we are beginning with projects that can be manged in '4 hour' chunks by one individual outside of a few exceptions to maintain social isolation. The exceptions are complex acute or sterile chronic experiments, which are done in full PPE.

  • Thanks 1

Share this post


Link to post
Share on other sites
Susan Tappan
1 minute ago, Lique Coolen said:

Hi Eline! Thank you for joining us. I have been using Zoom to meet with each of my lab members individually at least several times per week. We were able to share screens and look and work at data files together. It worked very well for us and I am even going to keep using it while we go back to the lab. Like you, I have many other responsibilities and need to work remotely. This way, my students can meet with me much more often and I can mentor them (even) better.

We've had success in my group also with having frequent all-hands meetings.  We do them daily, and it helps to have a time with no agenda, just a check-in to see how everyone is, what everyone is working on. We've learned from each other small tips and tricks about our remote set-ups, as well as can commiserate with slow networks etc.  There's new frustrations about working remotely, but some very real benefits too 🙂

  • Like 1

Share this post


Link to post
Share on other sites
Kip Ludwig
7 minutes ago, Guest eline van der beek said:

Just curious to learn how you maintain science discussion with the team or project based during remote working. Discussions are far from optimal when using the most common web based tools is my experience

Response here: My lab has had a lot of luck with Microsoft Teams.  We've even moved some of our Project Management/timelines to Microsoft Teams which facilitates 'off line' discussion and task management. 

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites
Lique Coolen
2 minutes ago, Mario Gil said:

I've been trying to figure out how to help facilitate/grant my students remote access to resources (software, server space) in the lab, but that has been difficult. And some work cannot be done from home due a number of issues (e.g., trainee's personal PCs don't have the memory required to run certain programs). Do you have any suggestions for neuroscientists who may not be as tech savvy as others. Are companies, NIH, others, starting to address this issue? Or perhaps there's software/programs available that I'm not aware of? 

Hi Mario. This is indeed a real concern. Several software companies have offered free trials, including several stats and graphics programs. Universities have been making deals also for such free trials. For example, my university offered a 30 day free trial for adobe products. As for computers; universities did deal with this differently and differently for their trainee populations. Unfortunately in the lab, we often have to purchase our won computers. I had encouraged all my trainees to take their desk top computers home and we used google drives to store all the files, saving on memory and hard drive space. Hope that helps.

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites
Mario Gil

Thanks, Lique, for your very helpful suggestions. Much appreciated. 

  • Like 1

Share this post


Link to post
Share on other sites
Kip Ludwig
6 minutes ago, Mario Gil said:

I've been trying to figure out how to help facilitate/grant my students remote access to resources (software, server space) in the lab, but that has been difficult. And some work cannot be done from home due a number of issues (e.g., trainee's personal PCs don't have the memory required to run certain programs). Do you have any suggestions for neuroscientists who may not be as tech savvy as others. Are companies, NIH, others, starting to address this issue? Or perhaps there's software/programs available that I'm not aware of? 

Response Here: As noted in a previous answer, we've been using Microsoft Teams in my lab, which also has project management capability to help us coordinate. We've been sharing code via Github, which does a great job of revision history with multiple authors. But a lot of this has been proactively managed by developing a structure with project teams that have a specific lead, and then works with the team to identify and implement the most useful software for their specific project.  I then meet with the team leads weekly via Microsoft Teams, and only plug into the project team meetings monthly.   

  • Like 1

Share this post


Link to post
Share on other sites
Lique Coolen
10 minutes ago, Guest Nikolay said:

Any advise for conducting research (specifically data collection) remotely? 

Data collection may not be possible for all types of research. But analysis of data or collection of data based on for raw data files, for example images or recordings, is possible.

  • Like 1

Share this post


Link to post
Share on other sites
Susan Tappan

Do you think this experience is changing how you design and propose your future experiments?

  • Like 1

Share this post


Link to post
Share on other sites
Guest Moo

Poor scientific rigor is not always malicious. Many times, I've seen myself and fellow researchers "cherry picking" data/excluding data that just seems off. What can we do to reduce that? Also, type 1 or type 2 error - which is more serious for pilot proof of concept studies?

Share this post


Link to post
Share on other sites
Mario Gil
Posted (edited)
15 minutes ago, Guest James Coughlan said:

Does anyone have pointers to resources that help researchers move away from paper-based records for human subjects experiments to electronic ones? For instance, what are the best practices for putting sensitive (possibly HIPAA-compliant but not necessarily) data on Dropbox or Google Drive, and sharing these data (when appropriate) with colleagues?

Hi James, I encourage you to consider using REDCap. If you don't have server space to run the program, the company does offer fee-based hosting services. It's a good data capture tool that's used in clinical research. I've used it before and it is effective. For more information, go to:  https://www.project-redcap.org/

 

You can also email a link to a survey to participants, and add collaborators as users so that they can also access your data. 

Edited by Mario Gil
  • Like 1

Share this post


Link to post
Share on other sites
Lique Coolen
11 minutes ago, Guest P Verghese said:

We conduct research that involves human participants, and are considering alternatives to in-person data collection. Our studies involve visual, auditory and tactile stimuli and we are considering alternate means to gather data remotely. Are their resources to provide guidance on the extra precautions that need to be taken during remote testing?

The best first step is your university IRB, as any modification to the studies with human participants will need to be approved. They will be able to help you with any extra precautions during remote testing.

  • Like 1

Share this post


Link to post
Share on other sites
Kip Ludwig
12 minutes ago, Guest James Coughlan said:

Does anyone have pointers to resources that help researchers move away from paper-based records for human subjects experiments to electronic ones? For instance, what are the best practices for putting sensitive (possibly HIPAA-compliant but not necessarily) data on Dropbox or Google Drive, and sharing these data (when appropriate) with colleagues?

Response Here: For HIPPA Compliant data, you can't use Dropbox/Google drive unless it has encrypted functionality, and that is usually facilitated by your Institution (and or your IRB). If it's not encrypted it has to be completely de-identified first, and the protocol for de-identification can vary from Institution to Institution. You'll need to talk to your local Institution to get a more clear answer on this one, given how it varies. 

  • Like 1

Share this post


Link to post
Share on other sites
Mario Gil

Thanks, Kip, for your very helpful suggestions. I will look into Microsoft Teams. 

  • Like 1

Share this post


Link to post
Share on other sites
Guest eline van der beek

Thanks for the suggestions: have one of you had the courage to start a new research topic, e.g. kick off a project with another lab or collaborator during the lock down or now during start up? How do you deal with on boarding new people when getting them started in the lab is difficult due to the social distance measures?

Share this post


Link to post
Share on other sites
Kip Ludwig
12 minutes ago, Guest Santani Teng said:

Some types of data collection can be moved to remote platforms, but not all. I would be curious to hear how folks approach the larger issues of reframing research questions to fit the available methods (eg. instead of a neurophysiological hypothesis, how would [X] process be reflected behaviorally), and how much a funder could/should be expected to accommodate that shift.

As an ex Program Director who still keeps close relationships with my NIH PD colleagues, they are very understanding and supportive of reframing research questions to fit available methods.  The important thing is to reach out to them by phone to let them know as soon as possible. NIH PDs are much more likely to be proactively helpful on the phone, as emails are FOIA-able and they have to stick to 'by the book' interpretations of policy (instead of work-arounds). If you communicate with them, they will go out of their way to be flexible. 

Share this post


Link to post
Share on other sites
Teresa Murray
4 minutes ago, Lique Coolen said:

Data collection may not be possible for all types of research. But analysis of data or collection of data based on for raw data files, for example images or recordings, is possible.

 

4 minutes ago, Susan Tappan said:

Do you think this experience is changing how you design and propose your future experiments?

This situation is prompting my lab and collaborators to acquire as much multidimensional data as possible. A few number of experiments, performed by one or two people, will then give us enough data for several other lab members to analyze. This also facilitates data analysis by persons who are unaware of treatment conditions.

Share this post


Link to post
Share on other sites
Lique Coolen
12 minutes ago, Guest Laura O'Dell said:

Thank you for doing this.  Are there materials available to provide for trainees on rigor and reproducibility?  Also, how do you suspect that journals/reviewers will be flexible when we are faced with the limitations in animals, resources, etc.  Again, thank you both for your time!  laura

 

Hi Laura. Here is a link to the SFN resources: https://neuronline.sfn.org/collection/resources-to-enhance-scientific-rigor

These training modules are outstanding and I use these all the time. You can view these with your trainees or incorporate these in courses. SFN is al working on new materials all the time. If you have specific request, please email me at any time.

As for the pressure to publish: The funders will be sensitive to reduced productivity, and be flexible in understanding that the publications will take a bit more time. 

  • Like 1

Share this post


Link to post
Share on other sites
Guest Brandon

Hello,

Does anyone have experience with mailing physical objects or materials to participants and viewing them interacting with those materials remotely as part of a study?

Here are some of the challenges I've thought of doing this method, and not all have ways to mitigate them:

1.       Participants opening and viewing the objects in the box before the study - I can tell participants they should not open the box before our session, and possibly design the study to not take into consideration familiarity with the objects, but what if a family member opens the box? Do I take the word of the participant they didn't view the objects beforehand?

2.       How does one account for environmental factors, such as lighting or noise in the participant's home? I can ask them to find a well-lit room or quiet room, but those are all relative and variable.

3.       Interruptions from family or children - What if I am doing a time-based task, and the participant is interrupted by a family member? If the task gets better with practice, is this something where I just drop their result, or do I ask them to redo the task?

4.       If they need to complete a physical object and mail it back for analysis, what happens if the object is damaged between their house and my house? - I can have them move the camera over the object before they send it back, but I won't have the object like I do for the other participants. Is this a case by case base where I figure out if a video is good enough or not?

Thanks,

Share this post


Link to post
Share on other sites
Kip Ludwig
16 minutes ago, Teresa Murray said:

What platforms do you recommend to monitor and discuss a wide variety of data? For example, my lab uses video and microscopy images, and electrophysiology and biosensor data to study live rodents in disease and injury models. It is cumbersome to review data in these diverse formats on Zoom.

Response here: Teresa - as you are aware we do very similar sorts of work.  We've had a lot of luck with Microsoft Teams in terms of being able to visualize multimodal data. We've been using GitHub as a shared repository with version control for code, and moved away from MatLab to Python to deal with MatLab processing limitations on 'laptops at home'. Matlab doesn't do parallel computing as well. 

Share this post


Link to post
Share on other sites
Maya Sapiurka
Posted (edited)
3 minutes ago, Lique Coolen said:

Hi Laura. Here is a link to the SFN resources: https://neuronline.sfn.org/collection/resources-to-enhance-scientific-rigor

These training modules are outstanding and I use these all the time. You can view these with your trainees or incorporate these in courses. SFN is al working on new materials all the time. If you have specific request, please email me at any time.

As for the pressure to publish: The funders will be sensitive to reduced productivity, and be flexible in understanding that the publications will take a bit more time. 

If there are new resources you would like to see created to support scientific rigor as a part of the FRN project, you can suggest them using this form: bit.ly/FRNinput

Edited by Maya Sapiurka

Share this post


Link to post
Share on other sites
Teresa Murray
Just now, Kip Ludwig said:

Response here: Teresa - as you are aware we do very similar sorts of work.  We've had a lot of luck with Microsoft Teams in terms of being able to visualize multimodal data. We've been using GitHub as a shared repository with version control for code, and moved away from MatLab to Python to deal with MatLab processing limitations on 'laptops at home'. Matlab doesn't do parallel computing as well. 

Thank you for the suggestion. MATLAB is one platform that is most problematic. Been thinking about Python for some time. Also, I will look into Microsoft Teams.

Share this post


Link to post
Share on other sites
Lique Coolen
9 minutes ago, Guest eline van der beek said:

Thanks for the suggestions: have one of you had the courage to start a new research topic, e.g. kick off a project with another lab or collaborator during the lock down or now during start up? How do you deal with on boarding new people when getting them started in the lab is difficult due to the social distance measures?

My university didn't allow for new studies to start, but we had just started a brand new technique in the lab in the weeks prior to the shut down. We were therefore allowed to continue the studies on this new research topic. The company supporting the technical approach were extremely accommodating and used virtual platforms to guide and instruct us; hence in person visits were replaced by virtual visits. It worked very well, although we sometimes had to use multiple platforms and cameras (phone, computer, etc) to get into all the details. I am now planning on using these virtual techniques as well for all on boarding of new personnel: we have four new people joining this summer, and in-person training is not possible. We are therefore recording videos by the experts in the lab, and use zoom/facetime to train. I will from now on have video resources for all our approaches; new trainees can view these before hand, making training more efficient. Hope this helps.

  • Like 1

Share this post


Link to post
Share on other sites
Kip Ludwig

 

5 minutes ago, Teresa Murray said:

 

This situation is prompting my lab and collaborators to acquire as much multidimensional data as possible. A few number of experiments, performed by one or two people, will then give us enough data for several other lab members to analyze. This also facilitates data analysis by persons who are unaware of treatment conditions.

Response here: Like Teresa, this has definitely changed the way we think about experiments going forward. We used the opportunity to map out our experiments and papers over the next year, develop and troubleshoot code for automated analysis and 'real time' visualization of key data to troubleshoot experiments, and most importantly to more fully embrace rigor. We are working towards putting all of our hypotheses online with proposed analyses on our website before an experiment has been performed, so we can be 'backtracked' to make sure we aren't doing retrospective analyses that would require different statistical methods than prospectively tested hypotheses.  

We've used the time to develop a multi-lab meeting to discuss experiments before they are conducted to get outside perspectives on how to refine/identify key confounds we might be missing via group think.  This has been insanely helpful, so we've also implemented this as we draft papers to make sure 'naive readers' can follow the story/figures before submitting the paper.   

We have found that the 'break' in between animal experiments has been a good thing to improve our pipeline and tools, so we are going to deliberately put more 'breaks' into the schedule to allow more time for data analysis in between experiments to refine procedure.  

Share this post


Link to post
Share on other sites
Lique Coolen
11 minutes ago, Guest Brandon said:

Hello,

Does anyone have experience with mailing physical objects or materials to participants and viewing them interacting with those materials remotely as part of a study?

Here are some of the challenges I've thought of doing this method, and not all have ways to mitigate them:

1.       Participants opening and viewing the objects in the box before the study - I can tell participants they should not open the box before our session, and possibly design the study to not take into consideration familiarity with the objects, but what if a family member opens the box? Do I take the word of the participant they didn't view the objects beforehand?

2.       How does one account for environmental factors, such as lighting or noise in the participant's home? I can ask them to find a well-lit room or quiet room, but those are all relative and variable.

3.       Interruptions from family or children - What if I am doing a time-based task, and the participant is interrupted by a family member? If the task gets better with practice, is this something where I just drop their result, or do I ask them to redo the task?

4.       If they need to complete a physical object and mail it back for analysis, what happens if the object is damaged between their house and my house? - I can have them move the camera over the object before they send it back, but I won't have the object like I do for the other participants. Is this a case by case base where I figure out if a video is good enough or not?

Thanks,

Hi Brandon. Your question is a bit difficult to answer as you seem to have something very specific in mind. I would recommend conducting a pilot study to  investigate which variables you may encounter and to establish an experimental protocol with instructions that can eliminate or mitigate the effects of such variables. Statistical analyses of the effects of the variables may be beneficial. My apologies that I can't be more helpful. Maybe other participants on the live chat have other ideas?

Share this post


Link to post
Share on other sites
Kip Ludwig
Posted (edited)
28 minutes ago, Guest Moo said:

Poor scientific rigor is not always malicious. Many times, I've seen myself and fellow researchers "cherry picking" data/excluding data that just seems off. What can we do to reduce that? Also, type 1 or type 2 error - which is more serious for pilot proof of concept studies?

Response here: My lab has used the virtual time to start moving our hypotheses and methods to our website 'before conducting the experiment' so we can be backtracked to ensure we aren't doing retrospective analyses.  We've also done a lot of research into statistics during the downtime, working with local statisticians to understand in cases where we want to do ask a new question using data already taken, how to do that appropriately. In many cases it's using 'half the data' to develop the hypothesis while reserving the remaining half to 'test the hypothesis prospectively'.  Most times we are just using retrospective analyses to 'identify a future hypothesis to be prospectively tested'. But the most important thing is to be transparent about all the analyses that you've performed, not just 'the ones that turned out'.    

I think for pilot proof of concept studies, both are equally serious. 

Edited by Kip Ludwig
Update with resources
  • Like 1

Share this post


Link to post
Share on other sites
Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...