Skip to content
This repository was archived by the owner on Nov 13, 2024. It is now read-only.
This repository was archived by the owner on Nov 13, 2024. It is now read-only.

Training speed #5732

@gauravlokha

Description

@gauravlokha

Using a gtx 1650 , I know not much at all but it's just for learning.
So usually I leave the training overnight. Usually I get around 350ms give or take and get around 2-3 iterations per second.
Now I am not using my pc since it's training but I get readings of 2900ms. But it's taking 2-3 seconds for each iteration. Can someone explain to me if I'm not getting something right?

update - when I started using my pc to post this, the training has gone back to 350ms and getting usual 2-3 iter per second.

== ==
== Current iteration: 0 ==
== ==
==-------------- Model Options ---------------==
== ==
== batch_size: 2 ==
== ==
==---------------- Running On ----------------==
== ==
== Device index: 0 ==
== Name: NVIDIA GeForce GTX 1650 ==
== VRAM: 2.85GB ==
== ==

Starting. Press "Enter" to stop training and save model.

Trying to do the first iteration. If an error occurs, reduce the model parameters.

!!!
Windows 10 users IMPORTANT notice. You should set this setting in order to work correctly.
https://i.imgur.com/B7cmDCB.jpg
!!!
[02:13:28][#2][0394ms][3.1017][3.3906]
[02:38:14][#3961][0365ms][0.6420][0.5605]
[03:03:14][#007982][0387ms][0.3743][0.3337]
[03:28:14][#011995][0378ms][0.3142][0.2822]
[03:53:14][#016011][0398ms][0.2847][0.2543]
[04:18:14][#020029][0374ms][0.2622][0.2386]
[04:43:14][#024026][0388ms][0.2474][0.2247]
[05:08:14][#028041][0405ms][0.2342][0.2157]
[05:33:14][#032056][0418ms][0.2255][0.2064]
[05:58:14][#036077][0403ms][0.2181][0.2011]
[06:23:14][#040095][0408ms][0.2109][0.1937]
[06:48:14][#044110][0383ms][0.2048][0.1901]
[07:13:14][#048125][0390ms][0.2010][0.1845]
[07:38:14][#052142][0405ms][0.1951][0.1816]
[08:03:14][#056151][0381ms][0.1910][0.1780]
[08:28:14][#060162][0384ms][0.1881][0.1751]
[08:53:14][#064181][0403ms][0.1851][0.1714]
[09:18:14][#068195][0366ms][0.1813][0.1682]
[09:43:14][#072208][0388ms][0.1792][0.1676]
[10:08:14][#076231][0386ms][0.1747][0.1653]
[10:33:14][#080247][0400ms][0.1739][0.1626]
[11:20:13][#082719][2986ms][0.1713][0.1590]
[11:23:17][#082798][2602ms][0.1662][0.1685]
[11:48:16][#083462][1489ms][0.1707][0.1607]
[12:13:00][#085904][0360ms][0.1333][0.1455]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions