Jump to content

All Activity

This stream auto-updates     

  1. Today
  2. Yesterday
  3. Last week
  4. Michael Oberdorfer

    Larry Brilliant on covid-19.

    Wired magazine features an interview with the epidemiologist Larry Brilliant on "How Well We Are Fighting Covid-19." Your thoughts?
  5. We received a lot of questions during this webinar and weren't able to answer them all live, but Bob and Kate wanted to get to as many possible and have answered more here! How can we advocate for estimation statistics when faced with pushback from proponents of traditional statistical tests? Bob: One strategy that can be effective is to point out that an effect size with a confidence interval contains all the information needed to conduct a statistical test, so by reporting them you are providing more information while still enabling the reader to conduct statistical tests against any desired null hypothesis. There are also some very good professional guidelines for reporting, many of which emphasize the importance of reporting and interpreting effects sizes and uncertainty (the APA, for example, suggests “wherever possible, base discussion and interpretation of results on point and interval estimates” (APA Publication Manual, 6th edition, p. 35). Perhaps Kate: I tend to report effect sizes and confidence intervals alongside my p-values as each provides its own information and then people looking for p-values are placated. The important thing is that effect sizes and confidence intervals are reported as these are important for others to be able to interpret your results and wishing you use your results to inform their own research. A p-value is the least informative of these pieces of information. I also encourage my students to report descriptive statistics such as means and standard deviations as these are really helpful for anyone wishing to use or replicate their research. When doing power analysis, sometimes one might want to estimate the effect size based on studies of similar phenomena. How similar is "similar enough"? This is a hard question to answer and as an expert in your field you may be best placed to judge. I would approach this by looking at the range of effect sizes published on a similar phenomenon giving more weight to the estimates from studies which seem closest to the one I'm planning. It’s also important to consider publication bias so recently I’ve started reducing published effect sizes by ⅓ in my sample size calculations to account for this. I would then run a series of sample size calculations based on the upper and lower effect estimates. In my sample size justification I would include a statement justifying my choice of effect size and if there were few relevant studies and I had to cast the net quite widely to find ‘similar’ studies then I would state this in my justification. The important thing is to be transparent in your reporting and then the reader/reviewer can follow your thinking. Could you address the question of power, pilot studies, and how to address the issues you mentioned in grant applications? There is a tension between current culture and the utility of pilot studies. Funders want to be sure the research they fund will be fruitful so encourage proof-of-concept or pilot data. This can be useful for showing the feasibility of the research procedure, a new method, or of being able to recruit relevant participants. What these small pilot studies are less informative for is estimating likely effect sizes as the results they produce will be imprecise and may be misleading. So I would focus on using the pilot study to evidence the feasibility of the study procedure, and then draw more widely on previously published effect sizes and data and/or minimal clinically important differences to base my sample size calculation on. Many top Journals publish so much normalized data it is impossible to get a feel for the data. How do we fix this? It is important to talk with editors and to advocate for clear and forward-looking policies on reporting and sharing data. It can be helpful to point out the NIH’s guidelines on pre-clinical reporting (https://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research). It can also be helpful to share policies from other journals, to help spur thought and discussion. For example, The Journal of Neurophysiology has extensively updated their author guidelines (and published accompanying tutorials/perspectives) for strong data transparency (https://journals.physiology.org/author-info.promoting-transparent-reporting ). Remember that editors and reviewers are often overtaxed and contributing essentially volunteer work to running the journal--so it can be helpful to partner and ally with them in helping to envision improved policies for reporting. How valuable can it be to discuss trends in data even if they miss statistical significance (happy to hear an answer from all) thank you all, this talk is all very interesting! Bob - I would argue that it is essential that we banish the ‘signficant’ vs. ‘non-significant’ dichotomy. We should report and interpret all of our results, countenancing both the effect observed in the sample and our uncertainty about generalizing. Just because a finding has 0 in its confidence interval does not mean we should ignore it (or worse) omit it from our manuscript (or even worse) claim it as proof of a negligible effect. By the same token just because a finding does *not* have 0 in its confidence interval does not mean we should take it as meaningful, certain, or established. Kate - I tend to follow the approach advocated here https://www.bmj.com/content/322/7280/226.1 and report the point estimate and confidence interval and the associated p-value (always the exact p-value unless it is very small p<0.0001). I then try and avoid using the words ‘statistical significance’ and instead focus on describing what the point estimate and confidence interval is telling me. To develop an initial estimate of sample size needed to achieve a certain power, can one use sequential statistical analysis to achieve an a priori defined significance level to efficiently identify a minimum sample size needed for power analysis? I think there are two things collated here. We can design a study based on a given effect size and calculate the sample size required to test that effect with a given alpha and power. This gives us certainty in planning and cost as we know how many participants or samples we will need. But if we are uncertain about the likely effect size and we can be flexible about how many participants we can test we could choose a different approach and opt for a sequential testing method where we recruit and test until we reach some predefined stopping criterion. This is often done in a Bayesian framework where repeated testing doesn’t incur a type I penalty. For example, we might set the criterion to a Bayes factor of 3 or ⅓ which could be interpreted as substantial evidence of your alternative hypothesis over the null or vice versa respectively. What are the best inferential statistics and descriptive statistics methods for describing your datasets analysis results characteristics of patient’s and controls biomarkers if not methods like p-values for clinicians? It can be helpful to think about how a statistic can be used by both clinicians and other researchers. In this regard a p-value is arguably the least informative statistic we can provide. It simply tells us whether there is statistical evidence against the null; e.g., is treatment A better than treatment B. A clinician might also be interested in the size of a treatment effect, so how much better is drug A than B as they may wish to weigh any gain in effectiveness against any side-effects and so on. So at a minimum I would always report the point estimates from my statistical test and the corresponding confidence intervals as this tells the clinician how precise the estimate of the treatment effect is. I would then think what other statistics I could use to put the effect into context. One example is Numbers Needed to Treat (NNT), that is the average number of patients who need to receive treatment A for one additional positive outcome. The closer it is to 1, the more effective the treatment is. For example, if the NNT for drug A compared with drug B for recovery from depression is 4, it means you have to treat 4 people with the drug A to get one additional recovery. As for descriptive statistics I always encourage reporting means and standard deviations for continuous variables, or for categorical variables, numbers in a given category and percent of overall sample. These descriptive statistics can be invaluable to other researchers, particularly for meta-analyses and sample size calculations. For people who do memory work with mice, would it be plausible to borrow effect sizes from human memory work? Probably not. If the measurements are on different scales, you would be relying on standardized effect sizes. One difficulty with standardized effect sizes, though, is that they are normalized to the variation observed in the study. This can make it challenging to compare standardized effect sizes across contexts, because not only the effect but also the variation could be different. For example, eye-blink conditioning in humans might be more variable than eye-blink conditioning in an inbred strain of mice, and this would make comparison of standardized effect sizes problematic. What if you want to report on a phenotype that you keep observing but having issues qualifying it- when is it okay to report observations- will there be scientist that completely disregard your data Is it always mandatory to use the "estimation method" comparing with the "frequentist"? Frequentist statistics is the philosophy of statistics that defines probability in terms of event counts and outcomes. A p value is a frequentist statistics. But so is a confidence interval. That is, frequentists can conduct tests *and* they can make estimates. So it’s not an either/or--it’s just how you want to apply frequentist statistics. When reviewing literature, what statistical information is most important to record? I'd like to collect relevant information the first time I read a paper rather than missing something important and having to dig through the study a second or third time. The Prisma Guidelines might be useful for helping you think through what you want to extract from each study as you work through a literature review: http://www.prisma-statement.org/ What is the difference between bootstrap and jackknife techniques? Are there advantages for bootstrapping over permutations testing? Is there a significant difference between bootstrapping and generating artificial data based on the data you have available? There is a lot of confusion over the names/procedures for resampling-based techniques. I like this taxonomy by Rodgers (1999) to help clarify the space of different resampling approaches: Wikipedia also has a pretty clear explanation of the different approaches: https://en.wikipedia.org/wiki/Resampling_(statistics) I’ve found Hesterberg (2015) to be the most clear and complete single source for understanding more about bootstrap techniques. Rodgers, J. L. (1999). The bootstrap, the jackknife, and the randomization test: A sampling taxonomy. Multivariate Behavioral Research, 34(4), 441–456. https://doi.org/10.1207/S15327906MBR3404_2 Hesterberg, T. C. (2015). What Teachers Should Know About the Bootstrap: Resampling in the Undergraduate Statistics Curriculum. American Statistician, 69(4), 371–386. https://doi.org/10.1080/00031305.2015.1089789 In neuroscience research, we can only afford to use small numbers of animals or slices, 4 pairs to 12 pairs at most. Therefore, the sample size is always small. Bayesian and bootstrap do not apply since they assume big sample size Changing statistical approaches cannot change the fact that, all else being equal, smaller samples provide less certain information than larger samples. The goal in research is to match the sample size to the research question--to obtain enough data to reliably generalize from your results. When you study patterns/trends that are abundantly clear and qualitative it can be possible to use relatively small sample sizes. There is not a firm definition of what counts as a qualitative effect, but a rule of thumb is that there would be essentially no overlap in the distributions between groups and that group means would be 3 or more standard deviations apart. With very very large and qualitative differences like this, it can be reasonable to use relatively small samples, though knowing that subtle differences and effects will not be resolvable. In general, the best proof that a sample size is reasonable is consistency of results across independent replications. If you can regularly obtain the same result with n = 6 across independent replications, then n = 6 is fine. If you see substantive fluctuations in results across independent replications, then either there are procedural problems or the sample size is not adequate. what is the best statistical approach when you have two populations unbalanced? In other words two population with a simple size very different between the two? Regarding Christophe's question about cultural change, what role should funding agencies play? Funders could have a tremendous influence, not only by setting standards for proposing and reporting research, but also by funding the training needed to improve rigor and reproducibility in science. For example, in the U.S. the NIH has a program to fund development of free training programs: https://www.nigms.nih.gov/training/pages/clearinghouse-for-training-modules-to-enhance-data-reproducibility.aspx Kate: Funders can be hugely influential in promoting culture change. What they say, we often do. Many funders in the UK have updated their funding application procedures to ask for evidence of reproducible research methods, and funders such as the MRC in conjunction with the NC3Rs offer training for their grant review panelists which focus on what to look for in a sound sample size justification. Do you know if reviewers in NIH or NSF panels are trained in Estimation Statistics? This could be a starting point. Bob: I’m not aware of any specific effort to ensure NIH or NSF panels know estimation statistics. Anecdotally, I’ve emphasized estimation in my recent neuroscience papers and NIH proposals and this has never been a problem. I do often mention how to interpret a confidence interval as a statistical test to help those who are focused on thinking only in terms of statistical significance. Why is an interval null superior? Could you comment on the important on null models when testing hypotheses? could you give an example of how one would use interval nulls? Many researchers who use hypothesis testing use a point null hypothesis. For example, in comparing the means of two groups, it is common to test against a null of exactly no difference (Hnull: Mdiff = 0). It is possible (though currently less common) to use an interval null--a range of values to be considered ‘essentially 0’. For example, we could compare the means of two groups using an interval null of +/- 10% (Hnull: -10% < Mdiff < +10%). Interval nulls are better: Point nulls are overwhelmed by sample size--with very large sample-sizes even extremely trivial differences will be flagged as statistically significant. Interval nulls allow the researcher to specify the range of effects that are trivial and to test only for effects that are clearly non-negligible. They work well even with very large sample sizes. Interval nulls unify statistical and practical significance - you know the old warning that ‘statistical significance does not mean practical significance’. That’s true when we test against an interval null because we’re only rejecting a single parameter value. But with an interval null, the test is significant only if the data is incompatible with the entire null interval--only if it is reasonable to rule out all negligible/trivial effects. Note that defining what counts as negligible/trivial can be a challenge, but that would still have been an issue when conducting a point-null test, and with an interval null one is challenged to think through and define this *before* seeing the results. Testing against point nulls provides no way to accept the null. A null of exactly 0.00000 is probably never true, and also there is no procedure with a point null to ‘accept’ the null. This makes for a poor set of decision-making tools for scientists. With an interval null, we get expanded options for testing-- when the entire CI is outside the interval null we have clear evidence the effect is meaningful (non-neglible), when the entire CI is inside the interval null we have clear evidence the effect is negligible, and when the CI is partially in the interval null we have an indeterminate result. So an interval null gives us 3 decision-making options: demonstrate meaningfulness, demonstrate negligible, or indeterminate test. Can you use the variability of a well-known method to predict the minimum effect size you can detect for a given sample size? Bob: Yes. If you have a good estimate of variability in a measure, you could use that to a) estimate the margin of error for any given sample of that measure and/or b) obtain power curves (expected power by effect size for a range of different sample sizes)
  6. Earlier
  7. until
    Neuroscience is stronger with diverse perspectives. Although there have been gains in the percentage of underrepresented neuroscience researchers in recent years, there is still more work to be done to increase representation of diverse researchers and to create inclusive and equitable research environments. On July 2nd, 12:00 – 1:00 pm EDT, SfN will host a panel discussion called, “Black Lives Matter and Neuroscience: Why this moment matters.” The discussion will be moderated by Joanne Berger-Sweeney, PhD, and will feature Nii Addy, PhD, Marguerite Matthews, PhD, and Fitzroy ‘Pablo’ Wickham. During the discussion, our panelists will speak about challenges diverse neuroscientists face within the field and provide guidance on how the neuroscience community can leverage this moment to influence change. Additionally, the panel will discuss the impact of COVID-19 on the tenure clock for junior faculty, and how that impacts faculty of color. Neuroscientists of all backgrounds are encouraged to attend and contribute to the discussion. Register for this event on Neuronline. This event is free and open, so please share with your neuroscience network outside of SfN.
  8. Welcome to the 2020 Roommate Matching Forum. Post here to coordinate housing for Neuroscience 2020. This Neuroscience 2020 Roommate Matching Forum is open to both SfN members and non-members. Don't forget to check out the rest of the Neuronline Community forum, to read blog posts, and join online discussions and events like live chats and open threads. All SfN members have access to the Community, where there are forums for every career stage, professional development, career advice, outreach, diversity, general discussion, and more. Browse Neuronline, an SfN website that helps neuroscientists advance in their training and career and connect with the global scientific community year-round. Anyone can access five resources every 30 days, while SfN members enjoy unlimited access, including participation in SfN webinars. If you're not a member, explore SfN's membership benefits and join to receive access today! --- Please note that this is a public forum, so use discretion when posting identifying information. Review this post for more information about maintaining privacy in the forum. If you prefer not to list your contact information here to connect with other users about sharing hotel accommodations, consider sending a private message instead. You can do this by clicking on the user you wish to contact from the post they have published or by posting and asking interested users to send you a private message. If you have any questions or concerns, email meetings@sfn.org or @Andrew Chen, Neuronline Community Specialist, at achen@sfn.org. Additionally, please read the attached 2020 SfN Roommate Matching Forum Disclaimer and Code of Conduct. 2020 SfN Roommate Matching Forum Disclaimer and Code of Conduct.pdf
  9. Andrew Chen

    ComSciCon at University of Virginia

    until
    ComSciCon at UVA is a science communication conference featuring skill-specific workshop. In today’s climate, the need for proper science communication is more important than ever. Science has expanded and made incredible strides towards improving our daily lives and health. However, the general public does not always trust scientific data, and many public policies are not based on science. This often stems from poor communication between scientists and the general public or policy makers. Scientists are often misunderstood because they cannot clearly articulate their message in simple, clear, jargon-free language. This workshop aims to cultivate the skills necessary for scientists to communicate complex research topics on a simple, effective level for the general public without compromising the quality of their research. Application Deadline: July 27, 2020 ComSciCon Date: August 17 - 20, 2020 Learn more about ComSciCon here: https://comscicon.com/comscicon-uva-2020-workshop
  10. For people working to implement remote psychophysical data collection, what are some ways you are controlling for things like display properties, the acoustic environment, etc.? I guess this wouldn't be pandemic-specific so much as remote-data specific, but we'll probably see a proportional uptick in data collected this way. E.g. Woods, Kevin JP, et al. "Headphone screening to facilitate web-based auditory experiments." Attention, Perception, & Psychophysics 79.7 (2017): 2064-2072. Tran, Michelle, et al. "Online recruitment and testing of infants with Mechanical Turk." Journal of experimental child psychology 156 (2017): 168-178.
  11. Thank you all so much for a fantastic discussion. Please keep posting your questions and comments. Kip and I will check this page regularly and continue to answer questions. Have a great rest of your day.
  12. Thank you for the resources. Since there are fewer results to discuss in our lab meetings, I will take the time to present and discuss these during lab meetings.
  13. Thank you for a great online discussion! Please continue to use this forum to continue the discussion, Lique and I will check back periodically over the next couple of days to answer any remaining questions.
  14. My collaborators and I have found ourselves taking more time to discuss our data analysis plans before collecting and analyzing data sets. Our collaborators have more time to meet, with more time to mull over procedures since they are not in the lab as often. We hope to make up for the extra time it takes to collect data by avoiding midstream changes in analyzing data after it is acquired. Of course, this should be the procedure any time. In retrospect, the pressure to analyze and publish may lead to jumping into non-optimal plans. This could lead to improved collaborations.
  15. Some resources on Neuronline to specifically address these details: https://neuronline.sfn.org/scientific-research/minimizing-bias-in-experimental-design-and-execution https://neuronline.sfn.org/scientific-research/ask-an-expert https://neuronline.sfn.org/collection/resources-to-enhance-scientific-rigor
  16. Response here: My lab has used the virtual time to start moving our hypotheses and methods to our website 'before conducting the experiment' so we can be backtracked to ensure we aren't doing retrospective analyses. We've also done a lot of research into statistics during the downtime, working with local statisticians to understand in cases where we want to do ask a new question using data already taken, how to do that appropriately. In many cases it's using 'half the data' to develop the hypothesis while reserving the remaining half to 'test the hypothesis prospectively'. Most times we are just using retrospective analyses to 'identify a future hypothesis to be prospectively tested'. But the most important thing is to be transparent about all the analyses that you've performed, not just 'the ones that turned out'. I think for pilot proof of concept studies, both are equally serious.
  17. Hi Brandon. Your question is a bit difficult to answer as you seem to have something very specific in mind. I would recommend conducting a pilot study to investigate which variables you may encounter and to establish an experimental protocol with instructions that can eliminate or mitigate the effects of such variables. Statistical analyses of the effects of the variables may be beneficial. My apologies that I can't be more helpful. Maybe other participants on the live chat have other ideas?
  18. Response here: Like Teresa, this has definitely changed the way we think about experiments going forward. We used the opportunity to map out our experiments and papers over the next year, develop and troubleshoot code for automated analysis and 'real time' visualization of key data to troubleshoot experiments, and most importantly to more fully embrace rigor. We are working towards putting all of our hypotheses online with proposed analyses on our website before an experiment has been performed, so we can be 'backtracked' to make sure we aren't doing retrospective analyses that would require different statistical methods than prospectively tested hypotheses. We've used the time to develop a multi-lab meeting to discuss experiments before they are conducted to get outside perspectives on how to refine/identify key confounds we might be missing via group think. This has been insanely helpful, so we've also implemented this as we draft papers to make sure 'naive readers' can follow the story/figures before submitting the paper. We have found that the 'break' in between animal experiments has been a good thing to improve our pipeline and tools, so we are going to deliberately put more 'breaks' into the schedule to allow more time for data analysis in between experiments to refine procedure.
  19. My university didn't allow for new studies to start, but we had just started a brand new technique in the lab in the weeks prior to the shut down. We were therefore allowed to continue the studies on this new research topic. The company supporting the technical approach were extremely accommodating and used virtual platforms to guide and instruct us; hence in person visits were replaced by virtual visits. It worked very well, although we sometimes had to use multiple platforms and cameras (phone, computer, etc) to get into all the details. I am now planning on using these virtual techniques as well for all on boarding of new personnel: we have four new people joining this summer, and in-person training is not possible. We are therefore recording videos by the experts in the lab, and use zoom/facetime to train. I will from now on have video resources for all our approaches; new trainees can view these before hand, making training more efficient. Hope this helps.
  20. Thank you for the suggestion. MATLAB is one platform that is most problematic. Been thinking about Python for some time. Also, I will look into Microsoft Teams.
  21. If there are new resources you would like to see created to support scientific rigor as a part of the FRN project, you can suggest them using this form: bit.ly/FRNinput
  22. Response here: Teresa - as you are aware we do very similar sorts of work. We've had a lot of luck with Microsoft Teams in terms of being able to visualize multimodal data. We've been using GitHub as a shared repository with version control for code, and moved away from MatLab to Python to deal with MatLab processing limitations on 'laptops at home'. Matlab doesn't do parallel computing as well.
  23. Hello, Does anyone have experience with mailing physical objects or materials to participants and viewing them interacting with those materials remotely as part of a study? Here are some of the challenges I've thought of doing this method, and not all have ways to mitigate them: 1. Participants opening and viewing the objects in the box before the study - I can tell participants they should not open the box before our session, and possibly design the study to not take into consideration familiarity with the objects, but what if a family member opens the box? Do I take the word of the participant they didn't view the objects beforehand? 2. How does one account for environmental factors, such as lighting or noise in the participant's home? I can ask them to find a well-lit room or quiet room, but those are all relative and variable. 3. Interruptions from family or children - What if I am doing a time-based task, and the participant is interrupted by a family member? If the task gets better with practice, is this something where I just drop their result, or do I ask them to redo the task? 4. If they need to complete a physical object and mail it back for analysis, what happens if the object is damaged between their house and my house? - I can have them move the camera over the object before they send it back, but I won't have the object like I do for the other participants. Is this a case by case base where I figure out if a video is good enough or not? Thanks,
  24. Hi Laura. Here is a link to the SFN resources: https://neuronline.sfn.org/collection/resources-to-enhance-scientific-rigor These training modules are outstanding and I use these all the time. You can view these with your trainees or incorporate these in courses. SFN is al working on new materials all the time. If you have specific request, please email me at any time. As for the pressure to publish: The funders will be sensitive to reduced productivity, and be flexible in understanding that the publications will take a bit more time.
  25. This situation is prompting my lab and collaborators to acquire as much multidimensional data as possible. A few number of experiments, performed by one or two people, will then give us enough data for several other lab members to analyze. This also facilitates data analysis by persons who are unaware of treatment conditions.
  26. As an ex Program Director who still keeps close relationships with my NIH PD colleagues, they are very understanding and supportive of reframing research questions to fit available methods. The important thing is to reach out to them by phone to let them know as soon as possible. NIH PDs are much more likely to be proactively helpful on the phone, as emails are FOIA-able and they have to stick to 'by the book' interpretations of policy (instead of work-arounds). If you communicate with them, they will go out of their way to be flexible.
  27. Thanks for the suggestions: have one of you had the courage to start a new research topic, e.g. kick off a project with another lab or collaborator during the lock down or now during start up? How do you deal with on boarding new people when getting them started in the lab is difficult due to the social distance measures?
  28. Thanks, Kip, for your very helpful suggestions. I will look into Microsoft Teams.
  29. Response Here: For HIPPA Compliant data, you can't use Dropbox/Google drive unless it has encrypted functionality, and that is usually facilitated by your Institution (and or your IRB). If it's not encrypted it has to be completely de-identified first, and the protocol for de-identification can vary from Institution to Institution. You'll need to talk to your local Institution to get a more clear answer on this one, given how it varies.
  1. Load more activity
×
×
  • Create New...