> For the complete documentation index, see [llms.txt](https://rbler1234.gitbook.io/mmscan-devkit-tutorial/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://rbler1234.gitbook.io/mmscan-devkit-tutorial/dataset.md).

# Dataset

### Usage

Initialize the dataset for a specific task with:

```python
from mmscan import MMScan

# (1) The dataset tool
my_dataset = MMScan(split='train'/'test'/'val', task='MMScan-VG'/'MMScan-QA')
# Access a specific sample
print(my_dataset[index])
```

### Data Access

Each dataset item is a dictionary containing key elements (**Bold text** represents key elements directly related to the task, while the rest represents auxiliary elements; **\[TS-V]** indicates that the element is visible in the test set, while other elements are invalid/not visible in the test set):

* **3D Modality**

  1. **"pcds"** (**\[TS-V]**, np.ndarray): Point cloud data with dimensions \[n\_points, 6(xyz+rgb)], representing the coordinates and color of each point.
  2. **"bboxes"** (dict): Information about bounding boxes within the scan, structured as { object ID: { "type": object type (str), "bbox": 9 DoF box (np.ndarray) }}
  3. "ori\_pcds" (**\[TS-V]**, tuple\[tensor]): Original point cloud data extracted from the .pth file.
  4. "instance\_labels" (np.ndarray): Instance ID assigned to each point in the point cloud.
  5. "class\_labels" (np.ndarray): Class IDs assigned to each point in the point cloud.

* **Language Modality**
  * Category and ID info
    1. **"sub\_class"**: The category of the sample.
    2. **"ID"**: The sample's ID.
    3. **"scan\_id"**: The scan's ID.
  * For Visual Grounding task
    1. **"target\_id"** (list\[int]): IDs of target objects.
    2. **"text"** (**\[TS-V]**, str): Text used for grounding.
    3. **"target"** (list\[str]): Text prompt to specify the target grounding object.
    4. "anchors" (list\[str]): Types of anchor objects.
    5. "anchor\_ids" (list\[int]): IDs of anchor objects.
    6. "tokens\_positive" (dict): Indices of positions where mentioned objects appear in the text.
  * For Question Answering task

    1. **"question"** (**\[TS-V]**, str): The text of the question.
    2. **"answers"** (list\[str]): List of possible answers.
    3. "object\_ids" (list\[int]): Object IDs referenced in the question.
    4. "object\_names" (list\[str]): Types of referenced objects.
    5. **"input\_bboxes\_id"** (**\[TS-V]**, list\[int]): IDs of input bounding boxes.
    6. **"input\_bboxes"** (**\[TS-V]**, list\[np.ndarray]): Input 9-DoF bounding boxes.

* **2D Modality**
  1. **'img\_path'** (**\[TS-V]**, str): File path to the RGB image.
  2. **'depth\_img\_path'** (**\[TS-V]**, str): File path to the depth image.
  3. **'intrinsic'** (**\[TS-V]**, np.ndarray): Intrinsic parameters of the camera for RGB images.
  4. **'depth\_intrinsic'** (**\[TS-V]**, np.ndarray): Intrinsic parameters of the camera for depth images.
  5. **'extrinsic'** (**\[TS-V]**, np.ndarray): Extrinsic parameters of the camera.
  6. 'visible\_instance\_id' (list): IDs of visible objects in the image.