§2024-12-14
$ wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2204-arm64-100.10.0.tgz <-- this is for arm64 machine
$ tar xvf src/mongodb-database-tools-ubuntu2204-arm64-100.10.0.tgz
$ sudo cp -v mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/* /usr/local/bin
[sudo] password for alexlai:
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/bsondump' -> '/usr/local/bin/bsondump'
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/mongodump' -> '/usr/local/bin/mongodump'
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/mongoexport' -> '/usr/local/bin/mongoexport'
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/mongofiles' -> '/usr/local/bin/mongofiles'
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/mongoimport' -> '/usr/local/bin/mongoimport'
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/mongorestore' -> '/usr/local/bin/mongorestore'
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/mongostat' -> '/usr/local/bin/mongostat'
'mongodb-database-tools-ubuntu2204-arm64-100.10.0/bin/mongotop' -> '/usr/local/bin/mongotop'
- mongoexport syntax:
- mongoexport --collection=
- mongoexport --collection=
- mongoexport Examples
$ wget https://downloads.mongodb.com/compass/mongosh-2.3.6-linux-arm64.tgz
$ tar xvf mongosh-2.3.6-linux-arm64.tgz
$ mongosh-2.3.6-linux-arm64/bin/mongosh --version
2.3.6
$ sudo cp mongosh-2.3.6-linux-arm64/bin/mongosh /usr/local/bin
$ mongosh --version
2.3.6
- mongoexport YusheiDB
(base) alexlai@JetsonOrinNano:~/R-proj$ mongoexport --uri="mongodb://siteRootAdmin:b23258585@ysmongov801.yushei.com.tw:27017,ysmongov802.yushei.com.tw:27017,ysmongov803.yushei.com.tw:27017/YuSheiDB?replicaSet=ys20241107&authSource=admin" --collection=car_in_out --out=car_in_out-`date +"%m-%d-%y-%H:%M"`.json --jsonArray
2024-12-14T19:21:03.379+0800 connected to: mongodb://[**REDACTED**]@ysmongov801.yushei.com.tw:27017,ysmongov802.yushei.com.tw:27017,ysmongov803.yushei.com.tw:27017/YuSheiDB?replicaSet=ys20241107&authSource=admin
2024-12-14T19:21:04.380+0800 [........................] YuSheiDB.car_in_out 0/491838 (0.0%)
2024-12-14T19:21:05.381+0800 [........................] YuSheiDB.car_in_out 8000/491838 (1.6%)
2024-12-14T19:21:06.380+0800 [#.......................] YuSheiDB.car_in_out 24000/491838 (4.9%)
2024-12-14T19:21:07.381+0800 [#.......................] YuSheiDB.car_in_out 40000/491838 (8.1%)
2024-12-14T19:21:08.380+0800 [##......................] YuSheiDB.car_in_out 56000/491838 (11.4%)
2024-12-14T19:21:09.380+0800 [###.....................] YuSheiDB.car_in_out 72000/491838 (14.6%)
2024-12-14T19:21:10.380+0800 [###.....................] YuSheiDB.car_in_out 72000/491838 (14.6%)
2024-12-14T19:21:11.380+0800 [###.....................] YuSheiDB.car_in_out 80000/491838 (16.3%)
2024-12-14T19:21:12.380+0800 [####....................] YuSheiDB.car_in_out 96000/491838 (19.5%)
2024-12-14T19:21:13.381+0800 [#####...................] YuSheiDB.car_in_out 112000/491838 (22.8%)
2024-12-14T19:21:14.380+0800 [######..................] YuSheiDB.car_in_out 128000/491838 (26.0%)
2024-12-14T19:21:15.380+0800 [#######.................] YuSheiDB.car_in_out 144000/491838 (29.3%)
2024-12-14T19:21:16.380+0800 [#######.................] YuSheiDB.car_in_out 152000/491838 (30.9%)
2024-12-14T19:21:17.380+0800 [#######.................] YuSheiDB.car_in_out 152000/491838 (30.9%)
2024-12-14T19:21:18.381+0800 [########................] YuSheiDB.car_in_out 168000/491838 (34.2%)
2024-12-14T19:21:19.380+0800 [########................] YuSheiDB.car_in_out 184000/491838 (37.4%)
2024-12-14T19:21:20.381+0800 [#########...............] YuSheiDB.car_in_out 200000/491838 (40.7%)
2024-12-14T19:21:21.380+0800 [##########..............] YuSheiDB.car_in_out 216000/491838 (43.9%)
2024-12-14T19:21:22.380+0800 [###########.............] YuSheiDB.car_in_out 232000/491838 (47.2%)
2024-12-14T19:21:23.380+0800 [###########.............] YuSheiDB.car_in_out 232000/491838 (47.2%)
2024-12-14T19:21:24.381+0800 [###########.............] YuSheiDB.car_in_out 240000/491838 (48.8%)
2024-12-14T19:21:25.380+0800 [############............] YuSheiDB.car_in_out 256000/491838 (52.0%)
2024-12-14T19:21:26.381+0800 [#############...........] YuSheiDB.car_in_out 280000/491838 (56.9%)
2024-12-14T19:21:27.380+0800 [##############..........] YuSheiDB.car_in_out 296000/491838 (60.2%)
2024-12-14T19:21:28.380+0800 [##############..........] YuSheiDB.car_in_out 304000/491838 (61.8%)
2024-12-14T19:21:29.380+0800 [##############..........] YuSheiDB.car_in_out 304000/491838 (61.8%)
2024-12-14T19:21:30.381+0800 [###############.........] YuSheiDB.car_in_out 320000/491838 (65.1%)
2024-12-14T19:21:31.381+0800 [################........] YuSheiDB.car_in_out 336000/491838 (68.3%)
2024-12-14T19:21:32.380+0800 [#################.......] YuSheiDB.car_in_out 352000/491838 (71.6%)
2024-12-14T19:21:33.380+0800 [#################.......] YuSheiDB.car_in_out 368000/491838 (74.8%)
2024-12-14T19:21:34.380+0800 [##################......] YuSheiDB.car_in_out 384000/491838 (78.1%)
2024-12-14T19:21:35.380+0800 [##################......] YuSheiDB.car_in_out 384000/491838 (78.1%)
2024-12-14T19:21:36.381+0800 [###################.....] YuSheiDB.car_in_out 392000/491838 (79.7%)
2024-12-14T19:21:37.381+0800 [####################....] YuSheiDB.car_in_out 416000/491838 (84.6%)
2024-12-14T19:21:38.380+0800 [#####################...] YuSheiDB.car_in_out 432000/491838 (87.8%)
2024-12-14T19:21:39.381+0800 [#####################...] YuSheiDB.car_in_out 448000/491838 (91.1%)
2024-12-14T19:21:40.380+0800 [######################..] YuSheiDB.car_in_out 456000/491838 (92.7%)
2024-12-14T19:21:41.381+0800 [######################..] YuSheiDB.car_in_out 464000/491838 (94.3%)
2024-12-14T19:21:42.380+0800 [#######################.] YuSheiDB.car_in_out 480000/491838 (97.6%)
2024-12-14T19:21:43.233+0800 [########################] YuSheiDB.car_in_out 491839/491838 (100.0%)
2024-12-14T19:21:43.233+0800 exported 491839 records
(base) alexlai@JetsonOrinNano:~/R-proj$ ls
car_in_out-2024-12-14-19:21.json
- mongoexport with --field and --query
- mongoexport --fieldFile=fields.txt to specify fields to be included
- mongoexport --queryFile=query.json
{"_id":{"$oid":"6715b7bff679da3f6e2a7f26"},"plateText":"BMH5561","inTime":"2024-03-31T21:41:05","outTime":"2024-03-31T23:42:37","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""},
{"_id":{"$oid":"675d86d2118e03ee070c2700"},"plateText":"MYH1959","inTime":"2024-12-14T18:14:49","carType":"motor","inCameraSource":"002","outTime":"2024-12-14T21:37:59","isExit":false,"isMonthlyCar":false,"isUserCreate":false},{"_id":{"$oid":"675d86e3118e03ee070c276e"},"plateText":"BDM9856","inTime":"2024-12-14T16:25:47","carType":"car","inCameraSource":"003","outTime":"2024-12-14T21:38:59","isExit":false,"isMonthlyCar":false,"isUserCreate":false},{"_id":{"$oid":"675d86ff118e03ee070c281f"},"plateText":"NXP0036","inTime":"2024-12-14T19:25:52","carType":"motor","inCameraSource":"002","outTime":"2024-12-14T21:38:59","isExit":false,"isMonthlyCar":false,"isUserCreate":false}]
To use mongoexport with --fieldFile=fields.txt --queryFile=query.json.
- fields.txt as
plateText
inTime
outTime
carType
inCameraSource
Exit
isMonthlyCar
isUserCreateout
CameraSource
- carQuery.json as,
{
"$and": [
{ "isExit": {"$eq": true}},
{ "carType": {"$eq": "car"}},
{ "isMonthlyCar": { "$eq": false}}
]
}
- motorQuery.json as,
{
"$and": [
{ "isExit": {"$eq": true}},
{ "carType": {"$eq": "motor"}},
{ "isMonthlyCar": { "$eq": false}}
]
}
mongoexport --uri="mongodb://siteRootAdmin:b23258585@ysmongov801.yushei.com.tw:27017,ysmongov802.yushei.com.tw:27017,ysmongov803.yushei.com.tw:27017/YuSheiDB?replicaSet=ys20241107&authSource=admin" --collection=car_in_out --out=car_in_out_car-date +"%m-%d-%y-%H:%M".json --fieldFile=fields.txt --queryFile=carQuery.json --jsonArray
mongoexport --uri="mongodb://siteRootAdmin:b23258585@ysmongov801.yushei.com.tw:27017,ysmongov802.yushei.com.tw:27017,ysmongov803.yushei.com.tw:27017/YuSheiDB?replicaSet=ys20241107&authSource=admin" --collection=car_in_out --out=car_in_out_motor-date +"%m-%d-%y-%H:%M".json --fieldFile=fields.txt --queryFile=motorQuery.json --jsonArray
(PY-3.10.15) alexlai@JetsonOrinNano:~/R-proj/car_in_out$ ls -l car_in_out_*
-rw-rw-r-- 1 alexlai alexlai 34583905 十二 15 09:04 car_in_out_car-12-15-24-09:04.json
-rw-rw-r-- 1 alexlai alexlai 74285869 十二 15 09:05 car_in_out_motor-12-15-24-09:04.json
(PY-3.10.15) alexlai@JetsonOrinNano:~/R-proj/car_in_out$ jq -s 'length' car_in_out_car-12-15-24-09\:04.json
142585
(PY-3.10.15) alexlai@JetsonOrinNano:~/R-proj/car_in_out$ jq -s 'length' car_in_out_motor-12-15-24-09\:04.json
303671
>>> 30 + 31 + 30 + 31 + 31 + 30 + 31 + 30 + 14
258
>>> 142585 / 258
552.6550387596899
>>> 303671 / 258
1177.0193798449613
2024-12-02 ti 2024-12-09
>>> (136+136+138+259+348+727+726)/7
352.85714285714283
>>> (502+428+512+738+876+1672+1498)/7
889.4285714285714
>>>
ChatGPT 4o mini
Log in
Sign up You said: This is my partial json file of a parking lot cars in and out records mongoexport from momgoDB.
{"_id":{"$oid":"6715b7bff679da3f6e2a7f26"},"plateText":"BMH5561","inTime":"2024-03-31T21:41:05","outTime":"2024-03-31T23:42:37","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7bff679da3f6e2a7f29"},"plateText":"RDT3105","inTime":"2024-03-31T21:36:40","outTime":"2024-03-31T23:37:08","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c0f679da3f6e2a8059"},"plateText":"BEW8761","inTime":"2024-03-31T19:37:27","outTime":"2024-03-31T21:54:59","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c0f679da3f6e2a816a"},"plateText":"ASQ6012","inTime":"2024-03-31T18:40:00","outTime":"2024-03-31T21:39:26","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c0f679da3f6e2a8219"},"plateText":"EBA5967","inTime":"2024-03-31T19:03:32","outTime":"2024-03-31T21:24:57","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c1f679da3f6e2a823e"},"plateText":"AJT0915","inTime":"2024-03-31T19:35:16","outTime":"2024-03-31T21:19:13","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c1f679da3f6e2a828a"},"plateText":"AJU1080","inTime":"2024-03-31T18:25:00","outTime":"2024-03-31T21:13:22","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c1f679da3f6e2a830c"},"plateText":"9801ZG","inTime":"2024-03-31T19:09:04","outTime":"2024-03-31T21:01:01","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c1f679da3f6e2a8331"},"plateText":"BRL9263","inTime":"2024-03-31T20:05:47","outTime":"2024-03-31T20:58:24","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c1f679da3f6e2a843a"},"plateText":"3676U8","inTime":"2024-03-31T17:27:17","outTime":"2024-03-31T20:34:37","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c1f679da3f6e2a8440"},"plateText":"BCM6155","inTime":"2024-03-31T18:09:59","outTime":"2024-03-31T20:33:02","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""} {"_id":{"$oid":"6715b7c2f679da3f6e2a84ca"},"plateText":"BVL1986","inTime":"2024-03-31T18:17:24","outTime":"2024-03-31T20:21:45","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""}
How to do use R to summaries the count by day. ChatGPT said: ChatGPT To summarize the count of cars by day using R, you'll need to:
Load your JSON data into R. Parse the inTime field into a date format. Group the data by the date part of the inTime. Count the number of records for each day. Here's how you can do it:
Step 1: Load the necessary libraries r Copy code library(jsonlite) library(dplyr) library(lubridate) jsonlite will help you read and parse your JSON data. dplyr provides functions to manipulate and summarize the data. lubridate helps you handle date and time. Step 2: Load and parse the JSON data If you have your JSON data saved as a file (e.g., parking_lot.json), you can read it using fromJSON() from the jsonlite package.
r Copy code
Read the JSON data from a file
data <- fromJSON("parking_lot.json", flatten = TRUE) If your data is in a string or directly available, you can use fromJSON() in a similar way.
Step 3: Convert inTime to a Date object Convert the inTime field to a date, extracting only the date part.
r Copy code
Convert 'inTime' to Date (without time)
data$inDate <- as.Date(data$inTime) Step 4: Summarize the count by day Now that you have the inDate field, you can group the data by day and count the occurrences of each day.
r Copy code
Summarize by day
summary_by_day <- data %>% group_by(inDate) %>% summarize(count = n()) %>% arrange(inDate)
View the summary
print(summary_by_day) This code will:
Group the data by inDate (which is the day part of inTime). Count how many records exist for each day (n()). Sort the results by date. Final Output The output summary_by_day will be a data frame showing the counts of cars entering the parking lot each day.
For example, it might look something like:
bash Copy code
A tibble: 3 × 2
inDate count
You said:
data <- fromJSON("car_in_out_car-12-15-24-09:04.json", flatten = TRUE) Error in parse_con(txt, bigint_as_char) : parse error: trailing garbage :false,"outCameraSource":""} {"_id":{"$oid":"6715b7bff679da3 (right here) ------^ ChatGPT said: ChatGPT The error you encountered is due to the fact that your JSON file contains multiple JSON records without a proper array wrapper. JSON objects in the file are not enclosed in an array, which makes it invalid for fromJSON() to parse directly.
Here’s how you can handle this issue in R:
Solution 1: Add Square Brackets to Make It a Valid JSON Array Your file appears to contain multiple JSON objects that should be in an array. To fix this:
Manually add square brackets [] around the JSON objects. Separate each object with a comma. For example, your JSON should look like this:
json Copy code [ {"_id":{"$oid":"6715b7bff679da3f6e2a7f26"},"plateText":"BMH5561","inTime":"2024-03-31T21:41:05","outTime":"2024-03-31T23:42:37","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""}, {"_id":{"$oid":"6715b7bff679da3f6e2a7f29"},"plateText":"RDT3105","inTime":"2024-03-31T21:36:40","outTime":"2024-03-31T23:37:08","carType":"car","inCameraSource":"","isExit":true,"isMonthlyCar":false,"isUserCreate":false,"outCameraSource":""}, ... ] Solution 2: Parse It with jsonlite Using a Different Approach If manually editing the JSON file is not feasible, you can also try reading the file and manually adding the array brackets in R:
r Copy code library(jsonlite)
Read the raw JSON data from the file
raw_data <- readLines("car_in_out_car-12-15-24-09:04.json")
Join the lines into one large string and add the square brackets
raw_json <- paste0("[", paste(raw_data, collapse = ","), "]")
Now parse the JSON string
data <- fromJSON(raw_json, flatten = TRUE) This will:
Read the raw file line by line. Combine the lines into a single string. Add square brackets around the entire string to form a valid JSON array. Parse it using fromJSON(). After Parsing Once the JSON is correctly loaded into data, you can proceed with the steps to manipulate and summarize the data as described earlier (converting inTime to a Date and counting occurrences by day).
You said:
View the summary
print(summary_by_day)
A tibble: 351 × 2
inDate count
ℹ 341 more rows
ℹ Use print(n = ...) to see more rows how to print more ???
ChatGPT said: ChatGPT To print more rows from a tibble in R, you can use the print() function with the n argument, which specifies how many rows you want to display.
Here’s how you can print more rows:
r Copy code
Print more rows (for example, 50 rows)
print(summary_by_day, n = 50) This will display the first 50 rows of the summary_by_day tibble.
If you want to print all rows: You can set n to a larger number, or simply display all rows if you need:
r Copy code
Print all rows (be careful if you have a large dataset)
print(summary_by_day, n = Inf) Alternatively, if the dataset is too large and you only need to see a few records for inspection, you can also use the head() or tail() functions to preview the first or last few rows:
r Copy code
View the first 10 rows
head(summary_by_day, 10)
View the last 10 rows
tail(summary_by_day, 10) These functions are helpful when you're working with large datasets and don't want to print the entire data at once.
ChatGPT can make mistakes. Check important info. ?