Skip to content

Commit d315ef0

Browse files
littlecabirialittlecabiria
committed
add figure
1 parent d35d5c5 commit d315ef0

2 files changed

Lines changed: 38 additions & 46 deletions

File tree

‎README.md‎

Lines changed: 38 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,83 +1,75 @@
1-
21
# BenchHub
32

43
<!-- badges: start -->
5-
<!-- badges: end -->
64

7-
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/benchhub_sticker.png" align="right" width="200">
5+
<!-- badges: end -->
86

7+
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/benchhub_sticker.png" align="right" width="200"/>
98

109
## Installation instruction
1110

12-
```r
11+
``` r
1312
devtools::install_github("SydneyBioX/BenchHub")
1413
```
1514

15+
## About
1616

17-
## About
17+
BenchHub is a data storage framework implemented in R language to facilitate living benchmarks. It aims to enhance reproducibility and accessibility of benchmarking studies by making it easier to store, analyse and share benchmarking data.
1818

19-
BenchHub is a data storage framework implemented in R language to facilitate living benchmarks. It aims to enhance reproducibility and accessibility of benchmarking studies by making it easier to store, analyse and share benchmarking data.
19+
The three components currently in BenchHub are:\
20+
- **Trio**: A data structure consisting of Dataset, Evidence and Metric to faciliate the sharing of benchmarking datasets within the community.\
21+
- **BenchmarkInsight**: A data structure for storing benchmarking results and provides a collection of visualisations to faciliate the analysis of benchmarking results. - **BenchmarkInsight**: A data structure for storing collections of Trio objects and helper functions for specific benchmarking tasks.
2022

21-
The three components currently in BenchHub are:
22-
- **Trio**: A data structure consisting of Dataset, Evidence and Metric to faciliate the sharing of benchmarking datasets within the community.
23-
- **BenchmarkInsight**: A data structure for storing benchmarking results and provides a collection of visualisations to faciliate the analysis of benchmarking results.
24-
- **BenchmarkInsight**: A data structure for storing collections of Trio objects and helper functions for specific benchmarking tasks.
23+
BenchHub aims to make benchmarking easier for multiple groups of users in the community: - **Benchmark Developers**: BenchHub offers a central storage for contributing benchmarking studies with the community - **Method Contributors**: BenchHub allows evaluation of new tools against established benchmarks in the database without starting from scratch\
24+
- **Benchmark Consumers**: BenchHub allows exploring and interpretation of results to make informed method selections
2525

26+
### Trio
2627

27-
BenchHub aims to make benchmarking easier for multiple groups of users in the community:
28-
- **Benchmark Developers**: BenchHub offers a central storage for contributing benchmarking studies with the community
29-
- **Method Contributors**: BenchHub allows evaluation of new tools against established benchmarks in the database without starting from scratch
30-
- **Benchmark Consumers**: BenchHub allows exploring and interpretation of results to make informed method selections
28+
Trio is built around three key components:
3129

30+
**Data**: Data used by the methods to generate output.\
31+
**Supporting Evidence**: Metadata to compare with the output of methods, such as cell type, patient outcome, disease pathway.\
32+
**Metric**: Evaluation metrics used to compare output of methods with supporting evidence.
3233

34+
Trio is implemented as R6 object with fields to store each of the components.
3335

34-
### Trio
36+
<p align="center">
3537

38+
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/triooverview.png" width="35%"/> <img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/triodatastructure.png" width="55%"/>
3639

37-
Trio is built around three key components:
40+
</p>
3841

39-
**Data**: Data used by the methods to generate output.
40-
**Supporting Evidence**: Metadata to compare with the output of methods, such as cell type, patient outcome, disease pathway.
41-
**Metric**: Evaluation metrics used to compare output of methods with supporting evidence.
42+
### BenchmarkInsight
4243

43-
44-
Trio is implemented as R6 object with fields to store each of the components.
45-
44+
BenchmarkInsight serves as a visualisation and analysis tool for benchmarking results. It contains multiple visualisation techniques to help researchers analyse benchmarking results in terms of data, methods and metrics.
4645

46+
Once results are evaluated using Trio, the output can be directly passed into the benchmarkInsight object. BenchmarkInsight currently supports the following plot types.
4747

4848
<p align="center">
49-
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/triooverview.png" width="35%" />
50-
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/triodatastructure.png" width="55%" />
51-
</p>
5249

50+
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/benchmarkinsight1.png" width="55%"/> <img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/benchmarkinsight2.png" width="35%"/>
5351

54-
### BenchmarkInsight
52+
</p>
5553

56-
BenchmarkInsight serves as a visualisation and analysis tool for benchmarking results. It contains multiple visualisation techniques to help researchers analyse benchmarking results in terms of data, methods and metrics.
54+
### BenchmarkStudy
5755

58-
59-
Once results are evaluated using Trio, the output can be directly passed into the benchmarkInsight object. BenchmarkInsight currently supports the following plot types.
60-
56+
Benchmarking often involve large amount of datasets and processing scripts. BenchmarkStudy serves as the organisation framework for such objects. It stores references to collections of Trio objects and helper functions that standardise method outputs into a common form to evaluation.
6157

6258
<p align="center">
63-
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/benchmarkinsight1.png" width="55%" />
64-
<img src="https://raw.githubusercontent.com/SydneyBioX/BenchHub/devel/inst/benchmarkinsight2.png" width="35%" />
65-
</p>
6659

67-
### BenchmarkStudy
60+
![](inst/benchmarkStudy.png){width="334"}
6861

69-
Benchmarking often involve large amount of datasets and processing scripts. BenchmarkStudy serves as the organisation framework for such objects. It stores references to collections of Trio objects and helper functions that standardise method outputs into a common form to evaluation.
70-
62+
</p>
7163

72-
## Vignettes
64+
## Vignettes
7365

74-
We provide a comprehensive list of vignettes for every key step of BenchHub.
75-
Please refer to the [website](https://sydneybiox.github.io/BenchHub/) to see all vignettes listed below:
66+
We provide a comprehensive list of vignettes for every key step of BenchHub.\
67+
Please refer to the [website](https://sydneybiox.github.io/BenchHub/) to see all vignettes listed below:
7668

77-
| Vignette | Description |
78-
|----------|-------------|
79-
| 1 [Introduction to the Trio Class](https://sydneybiox.github.io/BenchHub/articles/v01_intro_trio.html) | <ul><li>Setting up Trio objects</li><li>Adding supporting evidence</li><li>Defining evaluation metrics</li><li>Running evaluations</li></ul> |
80-
| 2 [Evaluation using TrioR](https://sydneybiox.github.io/BenchHub/articles/v02_Evaluation_using_Trio.html) | <ul><li>Cross-validation with consistent splits</li><li>Single-dataset evaluation</li><li>Exporting results to BenchmarkInsights</li></ul> |
81-
| 3 [Introduction to BenchmarkInsight Class](https://sydneybiox.github.io/BenchHub/articles/v03_intro_bmi.html) | <ul><li>Creating BenchmarkInsights objects</li><li>Visualization toolkit</li><li>Answering key benchmarking questions</li></ul> |
82-
| 4 [Uploading a Trio to Curated Trio Datasets](https://sydneybiox.github.io/BenchHub/articles/v04_uploading_a_Trio.html) | <ul><li>Creating GitHub Personal Access Token</li><li>Setting up googlesheets4 authentication</li><li>Uploading datasets to Figshare</li><li>Contributing Trios to the database</li></ul> |
83-
| 5 [Working with BenchmarkStudy](https://sydneybiox.github.io/BenchHub/articles/v05_benchmarkstudy.html) | <ul><li>Creating and organizing benchmark studies</li><li>Adding Trios and mapping functions</li><li>For benchmark developers: uploading studies</li><li>For method developers: evaluating new methods against existing benchmarks</li></ul> |
69+
| Vignette | Description |
70+
|-------------------------------------------------------------------------------------------------------------------------|-------------|
71+
| 1 [Introduction to the Trio Class](https://sydneybiox.github.io/BenchHub/articles/v01_intro_trio.html) | |
72+
| 2 [Evaluation using TrioR](https://sydneybiox.github.io/BenchHub/articles/v02_Evaluation_using_Trio.html) | |
73+
| 3 [Introduction to BenchmarkInsight Class](https://sydneybiox.github.io/BenchHub/articles/v03_intro_bmi.html) | |
74+
| 4 [Uploading a Trio to Curated Trio Datasets](https://sydneybiox.github.io/BenchHub/articles/v04_uploading_a_Trio.html) | |
75+
| 5 [Working with BenchmarkStudy](https://sydneybiox.github.io/BenchHub/articles/v05_benchmarkstudy.html) | |

‎inst/benchmarkStudy.png‎

177 KB
Loading

0 commit comments

Comments
 (0)